AI/ML Evaluation Specialist, Human Data

What you'll do

• Lead end-to-end human-data programs, including planning, delivery, and retrospectives.
• Develop measurement, quality-control, onboarding, and calibration frameworks for annotation workforces.
• Improve workflows and data pipelines, coordinate cross-functional partners, and ensure compliant data collection.

What they ask for

• Bachelor's degree or higher in Cognitive Science, Linguistics, or a related field with an experimental or empirical component.
• 4+ years leading cross-team AI/ML human-data programs involving annotation operations, quality frameworks, and evaluation strategies in an NLP/NLU or generative AI environment.
• Proficiency in Python, R, and SQL; hands-on experience designing and managing 0→1 human-in-the-loop data collection, annotation, and evaluation initiatives, including agentic workflows.
• Experience with diverse data types and multiple languages; expertise in annotation quality management, statistical process controls, and data quality metrics.

Nice to have

• Master's degree or higher in Cognitive Science, Linguistics, or a related field with an experimental or empirical component.
• 2+ years owning data strategy for frontier AI development and evaluation, including human alignment methodologies and agentic GenAI systems.

Pay

$144,600 and $263,800 USD base salary

Tools

Python, R, SQL, ETL

AI focus

Human-data operations and evaluation for multilingual, multimodal, and multi-turn AI features; applies human-centered AI and agentic workflows to improve generative AI quality, safety, reliability, and scalability.