ML Infrastructure Engineer - New Grad

What you'll do

• Implement RDU inference features, including speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference.
• Adapt vLLM and related serving frameworks to the RDU architecture, and build and maintain public inference API endpoints.
• Write accuracy and regression tests and debug latency, throughput, and reliability issues in production.

What they ask for

• B.S. in Computer Science, Electrical Engineering, or a related field.
• Strong programming skills in Python and understanding of algorithms, data structures, concurrency, and systems design.
• Coursework or project experience in distributed systems or machine learning.
• Experience with LLM serving frameworks (vLLM, SGLang, or TensorRT-LLM), inference techniques (speculative decoding, KV caching, or continuous batching), and PyTorch.

Nice to have

• M.S. or Ph.D. in a related field; API or backend service experience; internship or research experience in ML systems or backend infrastructure.
• Experience with disaggregated systems (prefill/decode) or open-source project ownership or contributions.

Pay

$131,200 — $196,800 USD (base salary)

Tools

Python, vLLM, SGLang, TensorRT-LLM, PyTorch

AI focus

Develop and serve LLM inference features on RDU accelerators, including speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference.