Anyscale
Distributed LLM Inference Engineer
$170K–$245K
San Francisco +1 · Hybrid · Full-time · Posted May 27, 2026
- Location
- San Francisco
Palo Alto
About the job
Build and optimize large-scale batch and online LLM inference solutions across Ray and LLM engines, focusing on performance, cost, and open-source contributions.
What you'll do
- Ship end-to-end batch and online inference solutions at high scale with product teams.
- Integrate Ray Data with LLM engines and optimize large-scale inference for low cost.
- Integrate and improve open-source software, following current research and community practices.
What they ask for
- Familiarity with large-scale ML inference, including high throughput and low latency.
- Familiarity with deep learning and frameworks such as PyTorch.
- Solid understanding of distributed systems and ML inference challenges.
Nice to have
- ML Systems knowledge or experience using Ray.
- Experience with vLLM or TensorRT-LLM communities, framework/compiler contributions, or GPUs/CUDA.
Pay
$170K – $245K (base salary)
Tools
Ray, Ray Data, vLLM, TensorRT-LLM, PyTorch, TensorFlow, Triton, TVM
AI focus
Develop and optimize distributed large-scale LLM and ML inference systems, including batch and online inference.
Model-written summary · the original posting is authoritative