Software Engineer - Inference Platform

What you'll do

• Build orchestration systems for distributed LLM inference, including routing, autoscaling, scheduling, and runtime management.
• Design and operate Model APIs with advanced inference capabilities and platform fundamentals such as authentication, quotas, and usage metering.
• Instrument, benchmark, debug, and improve production systems across Kubernetes, distributed runtimes, networking, and GPU workloads.

What they ask for

• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.
• Proven track record owning low-latency, reliable backend services, including rate limiting, auth, quotas, metering, and migrations.
• Experience with performance and reliability practices such as profiling, tracing, capacity planning, and SLO management.

Nice to have

• Experience with or contributions to LLM inference engines or frameworks such as vLLM, SGLang, TensorRT-LLM, TGI, or Dynamo.
• Deep Kubernetes experience, GPU workload operations, or experience with distributed scheduling, autoscaling, or service orchestration.

Pay

$180K – $360K USD base salary

Tools

Kubernetes, vLLM, SGLang, TensorRT-LLM, TGI, Dynamo

AI focus

Build and operate infrastructure and APIs for large-scale LLM inference, including structured outputs, tool/function calling, multimodal serving, and inference performance improvements.