AI Product Manager, AI Platform - Quality and Evaluation

What you'll do

• Set the AI Quality Platform strategy and roadmap, including LLM evaluation, benchmarking, and observability.
• Build standardized evaluation methodologies, datasets, benchmarks, and continuous testing and experimentation capabilities.
• Use quality metrics, customer feedback, and analytics to guide product decisions and coordinate delivery across teams.

What they ask for

• 5+ years of product management experience building technical platform, infrastructure, or developer-focused products.
• Demonstrated experience shipping production AI products powered by large language models.
• Experience defining products that use LLM evaluation frameworks, AI quality measurement, and continuous model evaluation.
• Deep understanding of offline and online evaluation, model benchmarking, regression testing, and AI quality metrics.

Nice to have

• Experience with retrieval-augmented generation (RAG), AI agents, MCP, or model orchestration frameworks.
• Experience with synthetic data generation, automated evaluation pipelines, internal developer platforms, or ML infrastructure products.

Pay

$136,000-$170,000 USD base salary; $149,500-$187,000 USD base salary for select locations.

Tools

Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo, RAG, MCP

AI focus

Build LLM evaluation, quality measurement, benchmarking, regression testing, experimentation, and observability capabilities for production AI systems.