Machine Learning Evaluation Engineer

What you'll do

• Build scalable evaluation pipelines, metrics, benchmark datasets, and regression tests for text, image, and video tasks.
• Analyze model failures and dataset coverage; develop failure taxonomies and automated analysis and reporting workflows.
• Collaborate with model teams and other partners to provide actionable evaluation findings for model iteration and fine-tuning.

What they ask for

• BS and a minimum of 3 years of relevant industry experience; 3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
• Strong knowledge of ML principles and statistical methods, and experience assessing dataset quality, diversity, representativeness, and coverage.
• Applied experience with generative models, prompt tuning, subjective evaluation, human-in-the-loop methods, and rigorous benchmark and metric design.
• Strong proficiency in Python and experience building automated evaluation pipelines; experience using AI-powered development and analysis tools.

Nice to have

• Theoretical and practical understanding of Computer Vision and Vision-Language Models.
• Strong written and verbal communication skills for cross-functional collaboration.

Pay

$150,400 and $277,600 USD base salary

Tools

Python, LLM-as-Judge, VLMs, LLM/VLM-as-a-Judge

AI focus

Evaluate multimodal foundation and generative models; use LLM-as-Judge and human grading, VLM-assisted dataset curation, and cloud-based LLM-powered workflows for failure analysis and reporting.