Technical Program Manager - Frontier AI Data Labeling

What you'll do

• Own end-to-end labeling and annotation programs, from scoping and task design through quality assurance and delivery.
• Translate model training and evaluation needs into labeling tasks, rubrics, and release benchmarks; build and calibrate expert rater teams.
• Track quality, throughput, and cost; manage vendors, tooling, dependencies, and timelines across teams.

What they ask for

• 5+ years in technical program management or data operations in an AI/ML or data-intensive environment; 3+ years considered for exceptional candidates.
• 2+ years hands-on with data labeling, annotation, or model evaluation, including writing guidelines and managing quality at scale.
• Experience sourcing and managing contractor or expert annotator workforces through marketplaces or labeling vendors.
• Working knowledge of annotation quality’s effect on model training, preference data, pairwise comparisons, and rater calibration (RLHF).

Nice to have

• Experience running or supporting RLHF or other training-data pipelines at an AI lab or applied ML team.
• Clinical, healthcare, regulated, or high-stakes data experience.

Tools

Polaris, Mercor, Upwork, Toloka, RLHF, Label Studio, Scale AI, Surge AI

AI focus

Runs labeling, preference and RLHF data, and evaluation programs that support healthcare AI model training, quality, and release decisions.