Prolific
Human Data Quality Engineer (Founding Team)
United Kingdom +1 · Full-time · Posted Jul 21, 2026
- Location
- United Kingdom
New York, New York, United States - Experience
- 5+ years building quality, evaluation, or annotation systems in AI, machine learning, LLMs, or human data environments.
About the job
Build quality frameworks and infrastructure for human-data programs, partnering with clients and engineering teams to improve data quality and evaluation for advanced AI models.
What you'll do
- Design quality frameworks and measurement systems, from project design and participant workflows to evaluation and launch readiness.
- Advise clients on data requirements and annotation schemas, and investigate quality issues to develop scalable improvements.
- Build automated checks, dashboards, monitoring, and quality infrastructure with product and engineering teams.
What they ask for
- Strong Python and SQL skills.
- Understanding of machine learning pipelines and how human data affects model performance.
- Strong analytical and statistical thinking, including experience designing scalable quality frameworks.
- Ability to influence stakeholders and translate data analysis and quality methods into buildable requirements.
Nice to have
- Experience with LLM evaluation, RLHF, AI safety, or red teaming.
- Experience with human annotation programs or human data operations.
Tools
Python, SQL
AI focus
Develop human-data quality and evaluation strategies and systems for advanced AI models, including work with frontier AI labs.
Model-written summary · the original posting is authoritative