Agent Evaluation and Self-Improvement Engineer

What you'll do

• Build cross-harness evaluations and automated checks; use production traces to create evaluation cases and improvement signals.
• Develop sandboxed self-improvement loops and realistic enterprise environments for testing agent changes and end-to-end tasks.
• Measure multi-agent performance and calibrate evaluators, while detecting cheating and unauthorized access.

What they ask for

• BS or MS in Computer Science, Engineering, or a related field (or equivalent experience).
• 12+ years building large-scale systems, including evaluation systems for ML or agents and their data pipelines.
• Hands-on experience building and evaluating agents on an open or closed agent harness; solid grounding in experiment design.
• Experience setting technical direction across teams as a senior individual contributor; proficiency in Python, Go, Rust, or similar.

Nice to have

• Built self-improving agent systems, or shipped an agent benchmark, evaluation framework, or simulated test environment adopted by other teams.
• Experience detecting reward hacking or eval gaming, building oversight for systems that modify themselves, or creating domain evaluations such as for chip design or hardware verification.

Pay

224,000 USD - 356,500 USD base salary (Level 5); 272,000 USD - 431,250 USD base salary (Level 6)

Tools

Python, Go, Rust

AI focus

Builds evaluation systems and self-improvement loops for autonomous agents, including agent benchmarks, multi-agent assessment, and safeguards against eval gaming.