Prolific
Human Data Quality Engineer (Founding Team)
North America +1 · Full-time · Posted Jan 12, 2026
- Location
- North America
New York, New York, United States - Experience
- 5+ years in building quality, evaluation, or annotation systems within AI, machine learning, LLMs, or human data environments.
About the job
Build quality systems and evaluation strategies for human data programs, partnering with clients and internal teams to ensure data meets the needs of advanced AI models.
What you'll do
- Design quality frameworks and build quality into human-data workflows from project design through launch and delivery.
- Partner with clients to define data and evaluation requirements, and collaborate with product and engineering to develop scalable quality infrastructure.
- Analyze quality issues and build measurement systems, automated checks, dashboards, and reporting.
What they ask for
- Strong Python and SQL skills.
- Solid understanding of machine-learning pipelines and how human data affects model performance.
- Strong analytical and statistical thinking, including experience designing scalable quality frameworks.
- Ability to engage and influence stakeholders and turn analysis and quality methods into buildable requirements.
Nice to have
- Experience with LLM evaluation, RLHF, AI safety, or red teaming.
- Experience with human annotation programs or data operations, or familiarity with evaluation methodologies such as calibration and inter-rater agreement.
Tools
Python, SQL
AI focus
Develop quality and evaluation systems for human data used to train and assess advanced AI models, including work in frontier AI domains.
Model-written summary · the original posting is authoritative