Harvey
Staff Software Engineer, Model Infrastructure
$231K–$340K
San Francisco · Hybrid · Staff · Full-time · Posted Sep 6, 2026
- Experience
- 10+ years of software engineering experience building large-scale distributed systems.
About the job
Leads development of Harvey’s model infrastructure platform, enabling reliable, scalable, observable, and efficient AI inference across model providers.
What you'll do
- Design and build model reliability, routing, provisioning, failover, and traffic-management systems.
- Integrate model providers and improve observability, capacity planning, and cost efficiency.
- Lead cross-functional technical initiatives and mentor engineers.
What they ask for
- 10+ years of software engineering experience building large-scale distributed systems.
- Experience designing and operating highly available production services.
- Strong programming skills in Go, Java, Python, Rust, or C++.
- Deep understanding of distributed systems, cloud infrastructure, networking, and observability; experience leading technical projects across engineering teams.
Nice to have
- Experience with AI infrastructure, LLM serving, or machine learning platforms.
- Experience with model routing, inference gateways, or policy-based serving systems.
Pay
$236,000 - $340,000 USD (base salary); $231K – $340K (base salary)
Tools
Go, Java, Python, Rust, C++, OpenAI, Anthropic, Azure OpenAI
AI focus
Builds infrastructure for AI inference, model routing and operations, provider integrations, model evaluation and deployment, and future training platforms.
Model-written summary · the original posting is authoritative