AI Engineer - Data and Evals

What you'll do

• Create evalsets, benchmarks, and agentic LLM verifiers using curated, production, synthetic, and failure data.
• Build scalable evaluation infrastructure, observability, experiment pipelines, and enterprise simulation environments.
• Analyze evaluation results and traces, then work with partner teams to improve product quality while balancing latency and cost.

What they ask for

• 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals.
• Experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evalset curation and synthetic data generation.
• Familiarity with evaluation methods, including deterministic metrics, LLM-as-a-judge, human labeling, and judge calibration.
• Experience building reliable evaluation or experiment infrastructure at scale, including CI/CD integrations and observability tooling.

Nice to have

• TypeScript.
• Experience with simulation environments and/or model post-training techniques.

Pay

$200,000 - $280,000 annually (USD, base salary)

Tools

Go, Python, TypeScript, SQL, OpenTelemetry, CI/CD

AI focus

Develop evaluation datasets, benchmarks, LLM verifiers, agent observability, and enterprise simulations to measure and improve AI assistants and agents; apply findings to agent harnesses, tool use, model routing, and task-specific model post-training.