Vals AI
Evaluations Engineer
$130K–$185K
San Francisco · On-site · Full-time · Posted Jun 27, 2026
- Experience
- Significant Python experience, especially in a professional setting; no number of years stated.
About the job
Evaluate newly released LLMs on Vals AI benchmarks, analyze performance and failure modes, and help publish results and maintain evaluation systems.
What you'll do
- Run benchmark evaluations on new LLM releases and work with foundation model labs on performance testing.
- Analyze model errors and patterns, and collaborate with the communications team to share findings.
- Maintain model integrations and help improve benchmark infrastructure.
What they ask for
- Familiarity with leading LLMs and their relative performance and practical use.
- Strong engineering fundamentals and a track record of building substantial projects.
- Significant professional experience with Python.
- Experience with development sprints, Git workflows, and pull request reviews.
Nice to have
- Experience benchmarking LLMs or creating benchmarks.
- Technical writing ability.
Pay
$130K – $185K (base salary)
Tools
Python, Docent, Django, React, AWS, CDK
AI focus
Evaluate LLMs on real-world task benchmarks, analyze their strengths, weaknesses, and failure modes, and maintain evaluation infrastructure.
Model-written summary · the original posting is authoritative