Jarmin
Member of Technical Staff (Evals)
San Francisco, CA, US · Staff · Full-time · Posted Aug 16, 2026
- Experience
- 3+ years of experience
- Visa sponsorship
- US citizen or visa holder only
About the job
Own systems for measuring AI Employee performance and improving evaluation, feedback, and failure detection over time.
What you'll do
- Build evaluation systems, including datasets, offline replay, scorers, and regression alerts.
- Develop feedback loops for models and agents and support autonomous failure analysis.
- Define and alert on good, degraded, and failed runs.
What they ask for
- Experience building evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality.
- Ability to turn ambiguous quality questions into concrete metrics and decisions.
- Strong software engineering fundamentals and experience shipping production systems.
- Willingness to work in person in San Francisco.
Nice to have
- Have shipped LLM systems to production.
- Strong data instincts, collaboration with researchers, or open-source contributions.
AI focus
Measure and improve AI Employee performance through evaluation systems, model and agent feedback loops, and failure analysis.
Model-written summary · the original posting is authoritative