Mercor
Member of Technical Staff, Enterprise Evals Platform
$220K–$425K
San Francisco · On-site · Staff · Full-time · Posted Aug 15, 2026
- Experience
- 1–5 years in industry-level software engineering; professional, academic, or research experience in agent engineering and evaluation.
- Education
- Bachelor’s Degree in Computer Science or a related technical degree.
About the job
Build and operate an evaluation platform that measures enterprise agents, identifies failures, and supports reliable agent improvements and rollout decisions.
What you'll do
- Define expert-informed golden sets and build verifiers for agent trajectories and outputs.
- Develop scalable evaluation environments, task suites, and grading infrastructure; analyze production failures and create regression tests.
- Optimize models, prompts, skills, and harnesses, and own rollout gates for agent changes.
What they ask for
- Professional, academic, or research experience in agent engineering and evaluation.
- Bachelor’s degree in Computer Science or a related technical degree.
- 1–5 years in industry-level software engineering.
- Experience building evaluation suites for LLM or agent systems, and familiarity with benchmark construction and gaming.
Nice to have
- Experience with Harbor environments and RL environments.
Pay
$220K – $425K (base salary)
Tools
LLM, terminal-bench, tau-bench, APEX, Harbor, RL
AI focus
Develop evaluation methods and infrastructure for AI agents, including expert-grounded rubrics, verifiers, failure analysis, and optimization across models, prompts, skills, and harnesses.
Model-written summary · the original posting is authoritative