Staff Machine Learning Engineer, Agent Eval Platform

What you'll do

• Design calibrated judges, rubrics, deterministic validators, and confidence-based human review for agent evaluations.
• Maintain calibration against human-labeled trajectories, investigate offline-to-online divergence, and update evaluation scenarios from production failures.
• Develop reward and process-reward signals to optimize agent behavior in simulation while validating success against end-state outcomes.

What they ask for

• 8+ years in applied ML, data science, or ML-adjacent engineering, with shipped work that is used.
• Experience turning subjective human judgement into a reliable measurement others can act on.
• Strong applied ML fundamentals, Python, and production-grade coding skills.
• Experience in at least 3 listed areas, including automated evaluation, human annotation, ranking or experimentation evaluation, model fine-tuning, reward modeling, agent trajectory analysis, or prompt engineering.

Tools

Python, LLMs, SFT, RLHF/RLAIF

AI focus

Evaluate and improve agentic AI through calibrated LLM judges, fine-tuned judge models, reward and process-reward modeling, and simulation-based optimization.