ServiceNow
Senior Machine Learning Engineer, Agent Eval Platform
Santa Clara, CALIFORNIA, United States · Senior · Full-time · Posted Aug 27, 2026
- Experience
- 5+ years in applied ML, data science, or ML-adjacent engineering; experience in at least 3 of the listed areas.
- Visa sponsorship
- For positions requiring access to controlled technology subject to export control regulations, employment is contingent on obtaining any required export license or other government approval.
About the job
Build the evaluation and reward-signal layer for enterprise AI agents, including calibrated judges that assess multi-step actions and support agent improvement.
What you'll do
- Design and calibrate trajectory judges using deterministic validators, LLM judges, and human-labeled examples.
- Develop reward and process-reward signals and use them to optimize agent prompts, tools, planning, retrieval, and routing in simulation.
- Maintain evaluation quality by investigating offline-to-online divergence and refreshing scenarios from production failures.
What they ask for
- 5+ years in applied ML, data science, or ML-adjacent engineering, with shipped work that was used.
- Experience translating subjective human judgment into a reliable measurement others can act on.
- Strong applied ML fundamentals and strong Python; ability to deliver production-grade code.
- Experience in at least 3 listed areas, including automated evaluation, human annotation, ranking or experimentation evaluation, model fine-tuning, reward modeling, agent trajectory analysis, or prompt engineering.
Tools
Python, LLMs, SFT, RLHF/RLAIF
AI focus
Evaluates and improves agentic AI through LLM-as-judge methods, judge fine-tuning, reward and process-reward modeling, and simulation-based agent optimization.
Model-written summary · the original posting is authoritative