ServiceNow
Staff Software Engineer, Agent Eval Platform
Santa Clara, CALIFORNIA, United States · Staff · Full-time · Posted Aug 20, 2026
- Experience
- 8+ years building production backend or infrastructure systems; experience in at least 3 of the listed technical areas.
About the job
Build the agent evaluation platform’s measurement and judgment layer, including infrastructure to run scenarios, observe agent behavior, simulate enterprise systems, and turn evaluation results into signals that improve agents.
What you'll do
- Build scalable orchestration for multi-turn agent evaluations, including scheduling, retries, isolation, reporting, and self-service execution.
- Develop agent observability and tracing, including trajectory data models, fault attribution, and cross-run comparisons.
- Build stateful enterprise-system simulations and help use evaluation signals to optimize agents.
What they ask for
- 8+ years building production backend or infrastructure systems.
- Strong in Python or Go (ideally both).
- Experience designing and operating systems that handle real traffic at scale.
- Experience in at least 3 listed areas: distributed systems; orchestration and workflow runtimes; observability internals; concurrent and async programming; data-intensive pipelines; or gRPC/protobuf service and interface design.
Tools
Python, Go, Temporal, Airflow, Argo, OpenTelemetry, gRPC, protobuf
AI focus
Evaluate agent trajectories using calibrated LLM judges and human labels, and use evaluation signals as reward signals to improve agent behavior.
Model-written summary · the original posting is authoritative