Innodata
Quality Lead, Agentic AI Workflow Evaluation
In Office - San Jose, California +1 · Remote · Full-time · Posted Sep 19, 2026
- Location
- In Office - San Jose, California
ISL - Remote - US - Experience
- 4+ years in quality assurance, quality management, or senior review work in annotation, evaluation, trust and safety, or a similarly judgment-intensive domain.
- Education
- Bachelor's degree or equivalent practical experience.
About the job
Own and operate the quality system for an onsite team evaluating complex agentic AI workflows, including audits, calibration, reviewer training, and delivery support.
What you'll do
- Design and run audits, sampling, scoring, quality measurement, and reporting; identify error patterns and quality trends.
- Lead calibration and rubric updates, and train and onboard reviewers using ramp criteria and production-readiness assessments.
- Support performance conversations, report to the customer alongside the Engagement Manager, and deputize for the Engagement Manager during absences.
What they ask for
- Bachelor's degree or equivalent practical experience.
- 4+ years in quality assurance, quality management, or senior review work in annotation, evaluation, trust and safety, or a similarly judgment-intensive domain.
- Direct experience owning a quality function, plus significant AI/ML evaluation experience.
- Hands-on familiarity with agentic systems; strong written communication; comfort with spreadsheets and a dashboarding tool; enough Python or SQL to pull and slice data.
Nice to have
- Experience training or onboarding reviewers into rubric-based work.
Pay
$75-85 p/hour (hourly salary)
Tools
Python, SQL
AI focus
Evaluates agentic AI workflows for safe task completion, respect for user intent and consent, and performance under scrutiny; the role builds and operates quality systems for this evaluation work.
Model-written summary · the original posting is authoritative