Tech Lead Manager, AI Evaluations

What you'll do

• Own the AI evaluation engineering roadmap and build scalable evaluation infrastructure, datasets, graders, analysis, and dashboards.
• Define quality measures and integrate reproducible evaluations into development, release decisions, and production monitoring.
• Lead and develop applied AI scientists and engineers, partnering across teams to improve products and customer outcomes.

What they ask for

• 10+ years of experience in software engineering, ML infrastructure, data systems, or a related technical field, including building and operating production systems.
• 5+ years of engineering or applied AI team management experience, including technical leadership and cross-functional delivery.
• Hands-on experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders.
• Strong coding ability in Python or another production language; understanding of evaluation methods, agent evaluation, and measurement quality.

Nice to have

• Experience with safety testing or red teaming.
• Familiarity with cloud infrastructure, Kubernetes, data stores, observability, or public benchmarks.

Pay

$250,000 - $350,000 annually (base salary)

Tools

Python, Kubernetes, LLM/GenAI

AI focus

Leads evaluation of LLM/GenAI products, including retrieval/RAG, assistants, and agents; builds evaluation infrastructure, model-based graders, quality measures, and feedback loops to improve reliability and product performance.