Senior Software Engineer, Inference

What you'll do

• Design and own serving components such as engine integration, batching, KV cache management, and quantized execution; improve latency and GPU throughput.
• Build distributed execution and orchestration capabilities, including parallelism, cache offload, GPU scheduling, routing, and autoscaling.
• Evaluate inference technologies, resolve customer issues, and contribute to reviews and team mentorship.

What they ask for

• Familiarity with LLM inference engines and ability to modify engine internals; strong knowledge of inference internals.
• Working knowledge of tensor and pipeline parallelism, NCCL, GPU memory, and interconnects.
• Advanced proficiency in Kubernetes platform architectures and strong programming proficiency in Go and Python; ability to debug and profile C++/CUDA.
• Excellent analytical, debugging, and problem-solving abilities.

Nice to have

• Upstream contributions to named inference projects or experience with disaggregated serving and KV cache offload at scale.
• Experience with RDMA or GPU isolation, or on-premises, air-gapped, or regulated enterprise software delivery.

Pay

USD 144,000 - 273,000 (base salary, Colorado); USD 137,000 - 315,000 (base salary, North Carolina & Texas)

Tools

vLLM, SGLang, TensorRT-LLM, TGI, NVIDIA NIM, Kubernetes, Go, Python

AI focus

Develop and optimize LLM inference and serving, including runtime efficiency, quantization, speculative decoding, mixture-of-experts serving, and distributed execution.