Senior Product Operations Manager, Evaluation Quality

What you'll do

• Define and maintain evaluation methodology, guidelines, templates, and quality checks for contract-attorney evaluations.
• Onboard, train, and calibrate raters; monitor their quality and maintain consistent judgment across markets and jurisdictions.
• Analyze evaluation results, diagnose errors, and deliver documented findings and dashboards to Product, Engineering, AI Research, and Applied Legal Research.

What they ask for

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
• Experience owning quality inputs for complex, expert-driven or human-in-the-loop work.
• Experience onboarding, training, and calibrating distributed expert raters, annotators, or reviewers.
• Experience analyzing quantitative and qualitative data, running or interpreting statistical tests, and communicating conclusions.

Nice to have

• Experience in legal tech or regulated industries; or across multiple markets, languages, or jurisdictions.
• Familiarity with LLM-as-judge / automated evaluation used alongside human eval.

Pay

$155,400 - $233,200 USD (base salary)

Tools

LLM-as-judge

AI focus

The role supports evaluation quality for Harvey’s AI products, including human evaluation and potentially LLM-as-judge or automated evaluation alongside human evaluation.