Town
AI Engineer, Evals & Agent Quality
$250K–$300K
San Francisco · On-site · Full-time · Posted Aug 29, 2026
About the job
Own the evaluation, quality measurement, and model-routing systems that assess and improve Town’s AI assistant across multi-step tasks and product surfaces.
What you'll do
- Build evaluation systems for assistant quality, including multi-step agent trajectories.
- Create golden datasets and labeling processes to validate improvements and catch regressions.
- Develop model routing and online evaluation tools, and work with product engineers to instrument quality and address issues.
What they ask for
- Hands-on experience building or owning LLM evaluation systems or large-scale offline/online quality measurement.
- Knowledge of evaluation tools and frameworks, and ability to reason about model routing tradeoffs.
- Ability to implement fixes, not only build dashboards and metrics.
- Senior- or staff-level engineering experience and comfort working in a greenfield environment.
Pay
USD 250K – USD 300K base salary
Tools
LLM
AI focus
Build evaluation and quality systems for an AI assistant, including trajectory measurement, datasets and labeling, model routing, and online evaluation.
Model-written summary · the original posting is authoritative