Inference
Senior Software Engineer - Model Performance
$220K–$320K
San Francisco · On-site · Senior · Full-time · Posted Jan 21, 2026
- Experience
- 2+ years in ML systems, inference optimization, or GPU programming.
About the job
Optimizes and productionizes the model inference stack to improve latency, throughput, cost efficiency, and support for new model architectures.
What you'll do
- Implement and productionize inference optimizations, including quantization, speculative decoding, KV cache optimization, continuous batching, and LoRA serving.
- Profile and improve inference frameworks, CUDA kernels, and GPU utilization; build performance benchmarks and tooling.
- Add and validate model architecture support, experiment with inference techniques, and collaborate with applied ML engineers.
What they ask for
- Strong proficiency in Python and familiarity with C++.
- Hands-on experience with LLM inference frameworks such as vLLM, SGLang, or TensorRT-LLM, or similar.
- Deep understanding of GPU architecture and experience profiling GPU workloads.
- Experience with PyTorch, LLM optimization techniques, and measurably improving system performance.
Nice to have
- Experience with CUDA programming.
- Familiarity with serving non-LLM models, distributed inference, open-source inference frameworks, Docker, or Kubernetes.
Pay
$220,000 - $320,000 USD base salary
Tools
Python, C++, vLLM, SGLang, TensorRT-LLM, CUDA, PyTorch, Docker
AI focus
Optimizes inference for language models and other models, including model serving, GPU performance, and support for new architectures.
Model-written summary · the original posting is authoritative