Aslan
Evals Lead
Washington, DC Metro Area · Full-time · Posted Aug 10, 2026
- Experience
- Must have owned evaluations for a shipped LLM or agent product and built a labeled dataset used for training; no years specified.
About the job
Owns quality evaluation for autonomous agents, operating the platform, building evaluation and simulation tools, and deciding whether builds are ready to ship.
What you'll do
- Operate the platform daily, document reproducible issues, and route model or harness problems to the right owners.
- Rate agent plans and build versioned datasets for regression testing and training; define checks and regression criteria across agent histories and deployments.
- Build a simulation harness to run a week of agent operation overnight against candidate builds.
What they ask for
- Strong Python.
- Production experience with an evaluation framework such as Inspect, Promptfoo, Braintrust, Arize, LangSmith, or equivalent.
- Familiarity with agent harness internals, including memory, context assembly, and tool selection.
- Experience with preference data, process supervision, or reward modeling; sufficient statistics knowledge to distinguish regression from noise on small samples.
Nice to have
- Experience evaluating agents running for days or weeks, or systems that cannot be rolled back or reset.
- Graph stores, provenance, or lineage tracking; eligibility for a US security clearance.
Tools
Python, Inspect, Promptfoo, Braintrust, Arize, LangSmith
AI focus
Evaluates autonomous LLM agents, including their decisions, performance over time, and training data; builds simulation and regression tooling.
Model-written summary · the original posting is authoritative