Sr. Applied Scientist, AI Evaluation & Quality Systems

What you'll do

• Design ground-truth generation pipelines and real-time monitoring for drift and quality degradation.
• Calibrate LLM evaluators against human-verified gold sets and build tools to analyze disagreements and improve annotator training and guidelines.
• Partner with ML teams, evaluator developers, and annotators; communicate findings to technical and non-technical stakeholders.

What they ask for

• 5+ years of industry experience in applied science or machine learning, including production-grade evaluation, annotation, or quality-assurance pipelines.
• Hands-on experience designing ground-truth pipelines across task types and annotation modalities, including cold-start scenarios.
• Experience building real-time monitoring or anomaly/drift detection systems for live data or ML pipelines; working knowledge of generative AI evaluation methodology.
• Proficiency in Python and relevant ML frameworks, production experience with LLM-based pipelines and agents, and an MS or PhD in a related quantitative field or equivalent practical experience.

Nice to have

• PhD in Computer Science, Machine Learning, Statistics, or a related field.
• Experience building configurable, extensible systems or tooling for practitioners.

Pay

$142,300 and $263,300 USD base salary

Tools

Python, LLM-as-a-Judge

AI focus

Develops and validates AI/ML evaluation systems, including LLM evaluators, human-verified ground truth, calibration, and quality monitoring.