LLM Serving Engineer (Cloud AI Engineering), Senior / Staff Engineer

What you'll do

• Develop scalable LLM inference platforms using serving, caching, parallelism, speculative decoding, model optimization, and kernel techniques.
• Develop and contribute to LLM serving packages, and work with customers and internal compiler, firmware, and platform teams on solutions.
• Improve serving efficiency through autoscaling, load balancing, routing, workload profiling, and optimization.

What they ask for

• Hands-on experience with one or more LLM serving or orchestration packages, such as Triton-Inference Server, vLLM, or SGLang.
• Strong Python and PyTorch development experience, with knowledge of transformer-based models and foundational LLMs.
• Strong computer science fundamentals, including algorithms, data structures, parallel and distributed programming.
• Bachelor’s degree and 4+ years, master’s degree and 3+ years, or PhD and 2+ years of related work experience.

Nice to have

• Open-source contributions to a GenAI package; experience architecting large-scale distributed systems.
• Kernel design experience (PyTorch, CUDA, Triton), knowledge of torch.compile or torchDynamo, or a PhD in a related field.

Pay

$162,000.00 - $243,000.00 USD base salary

Tools

Python, PyTorch, Triton-Inference Server, vLLM, SGLang, Ollama, KServe, CUDA

AI focus

Develop and optimize LLM inference serving, including model optimization, serving algorithms, specialized kernels, and inference acceleration.