Lead Product Data Scientist, Agents

What you'll do

• Develop agent measurement and evaluation frameworks covering quality, customer outcomes, costs, and revenue impact.
• Use experimentation and causal inference to assess agents’ effects on customer behavior, retention, and revenue.
• Set technical direction for agent analytics, mentor data scientists, and influence product and executive decisions.

What they ask for

• 10+ years in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production.
• Deep expertise in product measurement, experimentation, statistical modeling, and causal inference.
• Experience evaluating AI systems in production, including evaluation frameworks and human-labeling or LLM-as-judge methods.
• Proficiency in SQL and Python or R; experience developing production-quality analytical solutions and familiarity with dbt.

Nice to have

• Experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize.
• Experience with usage-based pricing, credit models, AI infrastructure costs, or LLM unit economics.

Pay

$172,000 — $258,000 USD base salary

Tools

SQL, Python, R, dbt, Braintrust, LangSmith, Arize

AI focus

Measure and evaluate AI agents, including their quality, task completion, customer outcomes, costs, and business impact; use coding agents and LLMs to accelerate analytics.