AIML - Sr Machine Learning Engineering Manager, Evaluation

What you'll do

• Build scalable model and agent evaluation systems, including benchmarks, LLM-based evaluators, simulation environments, and regression testing.
• Lead the evaluation flywheel from failure diagnosis through prompt, agent, data, and post-training refinement.
• Mentor engineers and set technical strategy while contributing to design, experimentation, implementation, and review.

What they ask for

• 8+ years of professional experience in machine learning, applied research, or software engineering, including production ML systems or large-scale experimentation platforms.
• 3+ years of technical leadership, including direct people management of machine learning or software engineers.
• Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
• Strong Python and ML engineering skills, with deep experience in LLMs or agentic systems and automated evaluation methods.

Nice to have

• Applying recent ML research to production systems or high-impact products.
• Experience with automatic prompt/context optimization, synthetic data, agent evaluation, or privacy-preserving/on-device ML.

Pay

$237,600 to $401,700 USD base salary

Tools

Python, LLM-based evaluators, large language models, reward models, reinforcement learning

AI focus

Foundation-model and agent evaluation, automatic optimization, synthetic data generation, and post-training refinement.