Cursor
Software Engineer, Model Routing & Inference
New York, NY +1 · On-site · Full-time · Posted Apr 7, 2026
- Location
- New York, NY
San Francisco, CA
About the job
Build and operate the inference platform that routes AI interactions, improving speed, reliability, and cost efficiency.
What you'll do
- Build an inference gateway that abstracts provider APIs and simplifies model onboarding.
- Design cross-provider failover to reduce user impact from provider outages.
- Design routing backpressure and admission control to manage traffic spikes.
What they ask for
- Deep experience building high-throughput, low-latency distributed systems, especially in inference serving, traffic routing, or real-time data pipelines.
- Comfort reasoning about cost/performance tradeoffs at scale, including GPU utilization, provider economics, and capacity planning.
- Strong software engineering fundamentals and experience shipping production systems that handle millions of requests.
- Ability to balance reliability, cost, latency, and user experience.
AI focus
Build the inference platform and routing systems that power AI interactions.
Model-written summary · the original posting is authoritative