Innodata
QA / Evaluation Lead
Hybrid - Washington D.C +1 · Remote · Full-time · Posted Jul 10, 2026
- Location
- Hybrid - Washington D.C
ISL - Remote - US - Experience
- 6+ years total professional experience, including 4+ years in data quality, evaluation methodology, or QA on AI/ML programs.
- Education
- Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
About the job
Owns quality and evaluation for a federal AI data platform by designing evaluation methods, repeatable validation processes, and evidence-based readiness measures.
What you'll do
- Design inter-annotator agreement methodology, including metrics, sampling, adjudication, and agreement thresholds.
- Define and operate evaluation frameworks and sampling-based quality control, including escalation routing and drift detection.
- Validate quality scoring and IAA computation, support model API evaluation, and document evaluation results.
What they ask for
- Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field; equivalent experience may substitute on a 2-for-1 basis.
- Hands-on expertise with Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha, plus AI/ML training data evaluation framework design.
- QC process design experience and Python skills for tooling, metric computation, and statistical analysis.
- Active Secret clearance with TS/SCI eligibility.
Nice to have
- Prior DoD or IC data quality program experience.
- CVAT or equivalent annotation platform QC workflow configuration; drift detection, model monitoring, or FMV/video annotation quality experience.
Pay
$45 to $50 p/hour (USD), hourly salary
Tools
Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, Python, SAM 2, Frontier model API, CVAT, Databricks
AI focus
Evaluates AI/ML training data and model outputs, including annotation quality, model performance, and drift; supports SAM 2 and Frontier model API validation.
Model-written summary · the original posting is authoritative