Senior Research Scientist, Model Evaluation

What you'll do

• Create ambitious evaluation benchmarks and translate model feedback into trustworthy, repeatable evaluations.
• Research LLM evaluation methods, including training LLM judges, refining LLM-based data synthesis pipelines, and improving evaluation efficiency.
• Build scalable, reusable tools for analyzing model performance.

What they ask for

• Strong software engineering skills.
• Experience rapidly prototyping LLM capability demonstrations and developing resources to measure capabilities.
• Dozens of hours reviewing complex data and LLM outputs to ensure high data quality.
• Rigorously measure AI capabilities and ensure evaluations align with the capabilities being assessed.

Tools

LLM

AI focus

Research and develop evaluation methods, benchmarks, LLM judges, and data synthesis pipelines to measure AI capabilities and model progress.