Evaluation Science Lead

What you'll do

• Set evaluation strategy and design statistically grounded methods and workflows combining human annotation and automated scoring.
• Translate evaluation results into recommendations and go/no-go evidence, and build performance monitoring and drift detection.
• Improve evaluation rubrics and practices with Quality Operations, Engineering, and other teams.

What they ask for

• 5+ years in evaluation science, data science, or ML systems development, including end-to-end ownership of evaluation systems at scale.
• Experience with statistical methods, measurement validity, annotator agreement, and annotation protocols.
• Proficiency with R, Python, and SQL; experience designing evaluation frameworks used by operational teams.
• Strong communication skills; discretion with confidential information; ability to work onsite and occasional evenings/weekends, with up to 10% travel.

Nice to have

• Master’s, PhD, or comparable experience in a related quantitative field.
• Experience evaluating generative AI, A/B testing, causal inference, experimental design, or applied linguistics.

Pay

$175,500 and $311,700 USD base salary

Tools

R, Python, SQL, autograders, LLM-as-judge

AI focus

Develops evaluation science for multilingual AI, including human and automated assessment, generative AI evaluation, and monitoring model quality across languages and markets.