Mercor
Research Scientist, APEX Benchmarks
$200K–$500K
San Francisco · On-site · Full-time · Posted Aug 27, 2026
- Education
- Ph.D. in machine learning, NLP, or a related field is preferred; equivalent industry or frontier lab research experience considered.
About the job
Leads the design, measurement, publication, and adoption of APEX benchmarks and expert-built datasets that assess frontier AI models on economically valuable professional work.
What you'll do
- Design benchmark tasks, taxonomies, scoring, difficulty calibration, contamination controls, and statistical methods.
- Build expert-produced datasets and grading rubrics; establish rigorous reporting and analyze benchmark results.
- Publish and communicate findings, collaborate with academic and industry partners, and work cross-functionally to scale benchmarks.
What they ask for
- Strong applied or academic research background in LLM evaluation, benchmarking, NLP, or a related field, with rigorous experimental design.
- Strong coding skills to build evaluation harnesses, run experiments, and analyze results.
- Statistical rigor in sampling, variance, contamination, and grader reliability.
- Clear technical and non-technical communication; comfort with ambiguity and interest in AI business and GTM.
Nice to have
- Ph.D. in machine learning, NLP, or a related field; equivalent industry or frontier lab research experience considered.
- Top-tier publications, public benchmark or dataset experience, frontier-lab evaluation experience, or relevant domain and rubric/annotation expertise.
Pay
$200K – $500K (base salary)
Tools
LLM
AI focus
Designs benchmarks and datasets to evaluate frontier models’ ability to perform professional work, including agentic work; develops scoring and measurement methods and analyzes model failures.
Model-written summary · the original posting is authoritative