Senior / Staff AI Engineer — Evals & Model Quality

What you'll do

• Build evaluation frameworks, datasets, benchmarks, graders, and regression suites for agent workflows and behavior.
• Integrate evaluations into development and releases; benchmark models and inform model selection.
• Convert production failures, feedback, and agent traces into evaluation cases and quality improvements.

What they ask for

• 5+ years of software engineering experience, including hands-on production AI/ML systems.
• Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
• Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
• Strong Python engineering skills and ability to build reliable production-grade systems.

Nice to have

• Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.
• Experience with open-weight models, fine-tuning, post-training, or model routing; B2B SaaS and 0-to-1 experience are also bonuses.

Pay

$300K – $350K (base salary)

Tools

Python

AI focus

Evaluate and improve AI agents, including multi-step behavior, tool use, model performance, and production reliability.