Senior System Software Engineer, Agentic Inference - Dynamo

What you'll do

• Build disaggregated and distributed inference capabilities across Dynamo-supported engines, including support for agentic workloads.
• Develop inference-state management and cache reuse/transfer to improve latency, throughput, GPU utilization, and costs.
• Build and evolve the distributed inference frontend, balancing scalability, request scheduling, latency, and throughput.

What they ask for

• Master’s or PhD, or equivalent experience.
• 10+ years in Computer Science, Computer Engineering, or a related field.
• Excellent Rust/Python programming and software design skills, including debugging, performance analysis, and test design.
• Understanding of modern LLM API semantics, including structured outputs, tool calling, reasoning controls, token accounting, context management, and multimodal inputs.

Nice to have

• Prior contributions to open-source AI inference frameworks such as vLLM, TensorRT-LLM, or SGLang.
• Experience optimizing GPU memory, KV and prefix caches, or high-performance networking for relevant workloads.

Pay

224,000 USD - 356,500 USD (base salary, Level 5); 272,000 USD - 431,250 USD (base salary, Level 6)

Tools

Rust, Python, vLLM, SGLang, TRT-LLM, TensorRT-LLM, NIXL

AI focus

Develop generative AI inference software for trained models on GPUs, including agentic inference with long-horizon reasoning, tool calling, and stateful multi-turn execution.