Speak
Assessment Design Lead
$100K–$140K
US - Remote · Remote · Full-time · Posted Aug 10, 2026
- Experience
- 4+ years, or equivalent depth in closely related psychometric/measurement work.
About the job
Leads the design and validation of Speak’s language assessments, defining proficiency constructs, rubrics, and quality standards and partnering with Product and ML on automated scoring.
What you'll do
- Define assessment goals, constructs, blueprints, rubrics, and scoring criteria across proficiency, placement, and curriculum mastery assessments.
- Own assessment validity, fairness, reliability, and sign-off, including validation studies and audits for bias or gaming.
- Partner with Product and ML Engineers on automated scoring, calibration, and feedback generation.
What they ask for
- 4+ years designing rubrics, blueprints, and item specifications for a shipped language assessment product, or equivalent depth in related psychometric/measurement work.
- Deep familiarity with language proficiency frameworks such as CEFR, ACTFL, or IELTS/TOEFL descriptors.
- Ability to assess fairness across learner populations and translate qualitative constructs into technical scoring requirements.
- Quantitative rigor in reliability, validity, and scoring statistics; AI fluency and judgment are also required.
Nice to have
- Speech or pronunciation science background; familiarity with adaptive testing or IRT.
- Advanced degree in psychometrics, measurement, applied linguistics, SLA, or a related quantitative field; assessment research or technical report experience.
Pay
$100K – $140K (base salary)
Tools
CEFR, ACTFL, IELTS/TOEFL, Cohen's/Fleiss' kappa, classical test theory, IRT
AI focus
Uses AI tools in assessment-design work and collaborates with ML Engineers on automated scoring, calibration, and feedback generation.
Model-written summary · the original posting is authoritative