Mercor
Research Engineer – Benchmarking
$200K–$500K
San Francisco · On-site · Full-time · Posted Aug 18, 2026
About the job
Own benchmarking pipelines, evaluation systems, and failure-analysis workflows to measure and improve frontier language models.
What you'll do
- Design and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning.
- Build end-to-end LLM evaluation systems, including runs, scoring, dashboards, and reporting.
- Analyze model failures and use findings to inform reward design, data curation, and benchmark improvements.
What they ask for
- Strong applied research background focused on model evaluation, benchmarking, and/or failure analysis.
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Solid grasp of data structures, algorithms, and backend systems.
- Comfort with APIs, SQL/NoSQL, and cloud platforms.
Nice to have
- Industry experience on a post-training or evaluation/benchmarking team.
- Publications at top-tier venues such as NeurIPS, ICML, or ACL, especially in evaluation or benchmarking.
Pay
$200K – $500K • Offers Equity (base salary or total pay not specified)
Tools
APIs, SQL/NoSQL
AI focus
Develop and run evaluations and failure analyses that guide training, post-training, RLVR, reward design, and data pipelines for frontier language models.
Model-written summary · the original posting is authoritative