Contra Labs
Special Project Lead
$150K–$200K
New York City (Onsite) · On-site · Full-time · Posted Jul 30, 2026
- Experience
- 3+ years designing studies, running evaluations, analyzing data, and presenting findings.
About the job
Leads client projects that create datasets, benchmarks, and evaluations of AI-generated creative and multimodal outputs, from research design through delivery and reporting.
What you'll do
- Design annotation protocols, rubrics, and evaluation methods; manage project scope, staffing, quality, timelines, and client relationships.
- Analyze results and present actionable findings through reports, workshops, case studies, and benchmark publications.
- Improve repeatable project processes and partner with engineering on supporting tools.
What they ask for
- Based in NYC and in office five days a week in Williamsburg.
- Experience with data labeling, annotation, human-in-the-loop systems, or AI/ML evaluation workflows.
- Track record managing several concurrent annotation, evaluation, or QA projects.
- A point of view on creative quality and what constitutes good creative work.
Nice to have
- Statistical evaluation methods, RLHF, LLM evaluation, or benchmark design.
- Research background in a related field, or equivalent depth from industry; ethnography, UX research, design-study methods, or startup experience.
Pay
$150,000 to $200,000 USD base salary
AI focus
Work includes post-training dataset creation, benchmarking, and evaluation of creative and multimodal AI outputs; experience with AI/ML evaluation workflows is requested.
Model-written summary · the original posting is authoritative