Nous Research
Machine Learning Engineer, Evals
Americas (US time zones) · Remote · Full-time · Posted Aug 13, 2026
- Experience
- 3+ years; evaluation experience may come from coursework, an internship, a side project, open source work, or a job.
About the job
Build and maintain agent capability evaluation systems, benchmarks, LLM-as-judge calibration, and failure-analysis workflows for researchers.
What you'll do
- Run and reproduce evaluation pipelines; build protocols to calibrate judges and track drift.
- Extend benchmarks with new tasks, environments, rubrics, graders, and QA.
- Analyze model failures and maintain recurring evaluation workflows and tooling.
What they ask for
- 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience.
- Experience with at least one LLM evaluation framework.
- Hands-on LLM experience; solid Python and familiarity with Git, CI/CD, Docker, and Linux command line.
- Understanding of basic evaluation statistics and at least 3 of the listed evaluation/benchmark competencies.
Nice to have
- RLVR / RLHF pipeline, training data curation, distributed eval orchestration, or benchmark design experience.
- Red teaming, adversarial evaluation, or familiarity with psychometrics or measurement theory.
Tools
Python, Harbor, Nemo Evaluator, Git, CI/CD, Docker, Linux, RAG
AI focus
Evaluating agent capabilities and LLM outputs; developing LLM-as-judge calibration, benchmarks, and failure analysis to inform model training and evaluation.
Model-written summary · the original posting is authoritative