Mercor
Research Engineer, Intern – Benchmarking
San Francisco · On-site · Internship · Posted Oct 27, 2026
About the job
Build and operate benchmarking, evaluation, and failure-analysis systems that guide improvements to frontier language models.
What you'll do
- Design and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning.
- Build end-to-end LLM evaluation systems, including scoring, dashboards, and reporting.
- Analyze model failures and create rubrics and evaluators to inform training, data, and benchmark improvements.
What they ask for
- Strong applied research background focused on model evaluation, benchmarking, and/or failure analysis.
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Solid grasp of data structures, algorithms, and backend systems.
- Comfort with APIs, SQL/NoSQL, and cloud platforms.
Nice to have
- Industry experience on a post-training or evaluation/benchmarking team.
- Publications at top-tier venues such as NeurIPS, ICML, or ACL, especially in evaluation or benchmarking.
Pay
$70 per hour (base salary)
Tools
APIs, SQL/NoSQL
AI focus
Evaluate tool use, agentic behavior, and real-world reasoning in language models; analyze failures and guide post-training, RLVR, reward design, and data pipelines.
Model-written summary · the original posting is authoritative