Hippocratic AI
LLM Inference Engineer (Mid, Sr, Staff)
Menlo Park, CA · On-site · Staff · Full-time · Posted Sep 30, 2026
About the job
Designs and optimizes distributed LLM inference serving systems to improve latency, throughput, cost, and reliability across production deployments.
What you'll do
- Design and implement multi-node and multi-LoRA serving architectures, including disaggregated serving and optimized caching.
- Apply quantization and speculative decoding techniques to improve inference performance while preserving model quality.
- Benchmark and optimize performance across deployment scenarios and GPU types.
What they ask for
- Experience optimizing LLM inference systems at scale.
- Proven expertise with distributed serving architectures for large language models.
- Hands-on experience implementing quantization techniques for transformer models and understanding of speculative decoding, including draft models and Eagle approaches.
- Proficiency in Python and C++, plus experience with CUDA programming and GPU optimization.
Nice to have
- Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM.
- Experience with custom CUDA kernels or production inference deployments.
Tools
LLM, FP4/FP6, Python, C++, CUDA, vLLM, SGLang, TensorRT-LLM
AI focus
Optimizes distributed LLM inference serving, including quantization, speculative decoding, caching, and multi-LoRA serving.
Model-written summary · the original posting is authoritative