Principal Software Engineer, Inference

What you'll do

• Own LLM serving architecture, including runtime integration, batching, KV cache, quantized execution, and distributed inference.
• Set performance and orchestration strategy for GPU scheduling, request routing, and autoscaling.
• Mentor engineers, lead architecture reviews, and present technical direction to business and executive audiences.

What they ask for

• Production experience modifying LLM inference engine internals.
• Expertise in inference internals, tensor and pipeline parallelism, NCCL, GPU memory and interconnects, and Kubernetes platform architecture.
• Strong Go and Python skills, with ability to read, debug, and profile C++/CUDA.
• Experience debugging or profiling multi-tier application workloads such as RAG and agents; excellent analytical and problem-solving abilities.

Nice to have

• Upstream contribution to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
• Experience with disaggregated inference, KV cache offload, RDMA technologies, GPU partitioning, or air-gapped enterprise software delivery.

Pay

USD 160,000 - 303,000 (base salary, Colorado); USD 152,000 - 349,000 (base salary, North Carolina & Texas)

Tools

vLLM, SGLang, TensorRT-LLM, TGI, NVIDIA NIM, Kubernetes, Go, Python

AI focus

Develop and optimize LLM inference and serving runtimes, including distributed execution, quantization, speculative decoding, and mixture-of-experts serving.