Senior Manager, Evaluation, Services Special Projects

What you'll do

• Set evaluation direction and own methods, datasets, rubrics, and LLM-based automated grading for assigned product layers.
• Lead, recruit, and mentor scientists and engineers; partner with engineering and applied ML teams to turn findings into product improvements.
• Present quality status, risks, and trade-offs to senior leadership and contribute evaluation results to release decisions.

What they ask for

• 6+ years in machine learning, data science, or machine learning evaluation, including 3+ years of direct people management.
• Track record hiring, developing, and leading applied science, data science, or machine learning engineering teams.
• Hands-on experience building and deploying LLM-based autograders and rubrics, including evaluation-set development, validation, and calibration.
• Experience evaluating conversational AI, search, recommendation, or ranking systems with offline metrics and online experimentation; strong statistics and experimental-design skills, including A/B testing.

Nice to have

• Experience evaluating agentic or tool-use systems, relevance and ranking, or groundedness and factuality.
• Experience with logging and instrumentation, synthetic evaluation data, third-party partner APIs, or privacy-preserving evaluation.

Tools

LLM

AI focus

Evaluates conversational AI, including request interpretation, retrieval and ranking, grounded response generation, and task execution through integrations; builds LLM-based autograders and rubrics.