Director, Evaluation and Quality Assurance, Services Special Projects

What you'll do

• Create automated grading and human evaluation programs, including metrics and evaluation processes.
• Integrate continuous evaluation results into release quality gates and prevent regressions across product and transaction workflows.
• Build and lead the organization, tooling, and operating processes while partnering with engineering, applied ML, and senior leadership.

What they ask for

• 12+ years in data science, machine learning evaluation, quality assurance, or test engineering, including leading multi-level organizations through managers.
• Experience building an evaluation or QA function from the ground up, including metrics, process, tooling, and organizational structure.
• Extensive experience with LLM-based autograders and rubrics, and with human evaluation methodology, logging, and A/B testing for consumer products at scale.
• BS in Computer Science, Engineering, or a related technical field, or equivalent experience.

Nice to have

• 15+ years in the stated fields, including leading multi-level QA or evaluation organizations through managers.
• Experience evaluating agentic or tool-use systems that take real-world actions.

Pay

USD $311,100 to $496,900 base salary

Tools

LLM-based autograders, A/B testing, third-party APIs

AI focus

Evaluation and quality assurance for conversational AI, recommendation, ranking, search, and potentially agentic or tool-use systems; includes LLM-based automated grading and human evaluation.