Senior Software Engineer, Private Cloud Compute Inference Routing

What you'll do

• Develop global traffic management, capacity-aware placement, load balancing, failover, admission control, and overload management.
• Build high-performance server software and network protocols; benchmark and tune routing strategies using operational metrics.
• Lead projects from architecture through production, collaborating with model serving, infrastructure, security, Siri AI, Apple Intelligence, and SRE teams.

What they ask for

• 3+ years designing, building, and operating large distributed systems in production.
• Server development in Java, C/C++, C#, Go, or Swift; experience with networking, client-server architectures, and routing, load balancing, or scheduling at scale.
• Knowledge of LLM inference serving, including request routing, KV-cache placement and reuse, prefill/decode disaggregation, and latency/throughput/accelerator tradeoffs.
• Experience benchmarking, debugging, and tuning distributed systems; Unix system internals; cloud deployment and observability technologies such as Kubernetes.

Nice to have

• Experience building LLM inference serving or routing systems such as llm-d, NVIDIA Dynamo, vLLM, or SGLang.
• Experience with secure protocols or cryptographic and privacy-preserving system design, such as TLS, X.509, Oblivious HTTP, or remote attestation.

Pay

USD $150,400 to $277,600 base salary; USD $142,300 to $263,300 base salary

Tools

Java, C/C++, C#, Go, Swift, Kubernetes, llm-d, NVIDIA Dynamo

AI focus

Develops and optimizes LLM inference routing and scheduling, including KV-cache-aware placement, prefill/decode disaggregation, and cache locality within Private Cloud Compute's privacy architecture.