Distyl AI
Applied AI Researcher, Benchmarking
$150K–$250K
San Francisco +1 · Hybrid · Full-time · Posted Oct 16, 2025
- Location
- San Francisco
New York
About the job
Designs evaluation frameworks and benchmarks to measure the reasoning, interaction quality, reliability, and real-world impact of intelligent systems.
What you'll do
- Build evaluation frameworks and benchmarks that reflect real-world complexity and measure system performance.
- Explore evaluation methods such as adversarial robustness testing, longitudinal tracking, and human-in-the-loop assessment.
- Develop rigorous approaches to quantify capability and inform research priorities and industry standards.
What they ask for
- Experience building or maintaining benchmarks, test suites, or experimental frameworks for model or system performance.
- Statistical and analytical rigor, including designing fair, reproducible experiments and interpreting noisy results.
- Strong programming and data analysis skills to prototype ideas and run experiments.
- A demonstrated record of research results.
Nice to have
- Expertise in compound AI systems, agentic collaboration, and techniques such as ensembling, ReAct, or graph-of-thoughts.
- Daily use of AI tools such as ChatGPT, Cursor, and Perplexity.
Pay
USD $150K – $250K base salary
Tools
ChatGPT, Cursor, Perplexity, ReAct
AI focus
Evaluates intelligent systems and develops benchmarks for their capabilities, robustness, reliability, and operational impact; builds with models rather than training or fine-tuning them.
Model-written summary · the original posting is authoritative