Microsoft
Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)
$143K–$304K
Redmond, WA · On-site · Principal · Full-time · Posted Oct 7, 2026
- Experience
- Required: 6+ years with a bachelor’s degree, 4+ years with a master’s degree, or 3+ years with a doctorate in a listed or related field; equivalent experience accepted. Preferred: 9+ years with a master’s degree or 6+ years with a doctorate; equivalent experience accepted. Additional preferred experience includes 5+ years in publications, 2+ years as an invited speaker, 5+ years conducting research, and 3+ years developing and deploying production systems.
- Education
- Bachelor’s, master’s, or doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field; or equivalent experience.
About the job
Defines and builds AI evaluation systems to measure, improve, and responsibly ship LLM, RAG, and agentic products throughout their lifecycle.
What you'll do
- Architect evaluation frameworks and integrate evaluations into development workflows, build pipelines, release gates, and post-launch monitoring.
- Analyze agent behavior, quality gaps, and failure modes across models, retrieval, orchestration, tools, and product experiences.
- Lead cross-functional technical plans and mentor engineers and applied scientists building reusable evaluation infrastructure.
What they ask for
- Bachelor’s degree in a listed or related field and 6+ years of related experience; or a master’s and 4+ years; or a doctorate and 3+ years; or equivalent experience.
- Demonstrated experience evaluating LLMs, including creating evaluation datasets, defining quality metrics, analyzing behavior, and using results to guide improvements.
- Experience architecting complex systems and bringing AI, ML, or applied-science methods into production systems and product codebases.
- Technical coding and design skills, with ability to lead cross-functional initiatives and communicate effectively across teams.
Nice to have
- Master’s degree and 9+ years of related experience; or doctorate and 6+ years; or equivalent experience.
- Experience with production AI products, publications, invited conference talks, research programs, or evaluation infrastructure.
Pay
USD $142,800 - $274,800 per year (base pay); USD $188,000 - $304,200 per year (base pay)
Tools
LLM, RAG, ML, DS, CI/CD, Copilot
AI focus
Evaluates LLMs, RAG, and agentic AI systems across the product lifecycle, including behavior, retrieval, safety, and quality; integrates evaluation into product development and release decisions.
Model-written summary · the original posting is authoritative