Lilt
Senior Quality Manager, Applied AI
Boston, MA +5 · Hybrid · Manager · Full-time · Posted Sep 1, 2026
- Location
- Boston, MA
Washington D.C.
Indianapolis, IN
New York, NY
London, UK
San Francisco, CA - Experience
- 6+ years building or operating quality programs for ML/LLM systems.
- Education
- B.S./M.S. in a quantitative field (CS, Statistics, Data Science, Engineering, Linguistics, HCI) or equivalent practical experience.
- Visa sponsorship
- Authorization to work in the U.S. is a precondition of employment.
About the job
Leads quality systems, measurement, tooling, and customer remediation for Applied AI programs, ensuring deliverables meet documented quality targets before shipping.
What you'll do
- Design and operate quality workflows, metrics, and automated infrastructure across concurrent programs.
- Manage operational quality, including go/no-go sign-off, quality remediation, and process improvements.
- Develop rater reliability, calibration, adjudication, and integrity practices for human evaluation programs.
What they ask for
- 6+ years building or operating quality programs for ML/LLM systems.
- Experience designing end-to-end quality frameworks across multiple concurrent programs.
- Strong knowledge of reliability and agreement methods, statistics, error analysis, and executive reporting.
- Hands-on experience with labeling/evaluation tooling and deterministic validations; understanding of dataset governance and audit-ready documentation.
Nice to have
- Fluency in multiple human languages.
- Proficiency in a programming language and experience with modern AI frameworks.
Tools
ML/LLM systems, Label Studio, Krippendorff’s α, Cohen’s κ
AI focus
Quality evaluation and human-in-the-loop operations for ML/LLM systems, including labeling, benchmarks, model evaluation, and quality instrumentation.
Model-written summary · the original posting is authoritative