ServiceNow
Staff Software Engineer, Agent Eval Platform
Mountain View, CALIFORNIA, United States · Staff · Full-time · Posted Aug 26, 2026
- Experience
- 8+ years building production backend or infrastructure systems; experience in at least 3 of the listed technical areas.
About the job
Build the evaluation platform’s judgment layer and infrastructure to score multi-step AI-agent actions, improve agents using evaluation signals, and support reliable testing in stateful enterprise environments.
What you'll do
- Develop and calibrate evaluation rubrics and judges against human labels, with signals that can help train agents.
- Build scalable orchestration, observability and tracing, and stateful simulation systems for agent evaluations.
- Establish reliable, repeatable evaluation practices, including run comparison, fault attribution, and self-serve access for teams.
What they ask for
- 8+ years building production backend or infrastructure systems.
- Strong in Python or Go (ideally both).
- Experience designing and operating systems that handle real traffic at scale.
- Experience in at least 3 listed areas: distributed systems; orchestration and workflow runtimes; observability internals; concurrent and async programming; data-intensive pipelines; or gRPC/protobuf service and interface design.
Tools
Python, Go, Temporal, Airflow, Argo, OpenTelemetry, gRPC, protobuf
AI focus
Build and calibrate LLM-based evaluation judges for agent trajectories, and develop evaluation signals that can be used to optimize and train agents.
Model-written summary · the original posting is authoritative