Member of Technical Staff, Inference & Serving

What you'll do

• Build high-performance, low-latency serving systems and extend orchestration for distributed inference, evaluation, and large-batch serving.
• Implement endpoint load balancing, autoscaling, traffic routing, model versioning, canary deployments, and zero-downtime rollouts.
• Develop monitoring and observability tools, and work with ML researchers to productionize model and serving improvements.

What they ask for

• Knowledge of ML serving frameworks such as SGLang, vLLM, Triton Inference Server, or TensorRT-LLM.
• Understanding of PyTorch or TensorFlow from a systems perspective and familiarity with CUDA and high-performance computing.
• Experience with Docker, Kubernetes, CI/CD pipelines, and performance optimization and profiling of ML systems.

Nice to have

• Experience building or maintaining language models with tens of billions of parameters or more.
• Experience with distributed systems, cloud platforms, ML workflow orchestration, model optimization, or ML infrastructure challenges.

Tools

Kubernetes, Ray, SLURM, SGLang, vLLM, Triton Inference Server, TensorRT-LLM, PyTorch

AI focus

Production inference and serving for diffusion LLMs, including applying model advances such as new architectures, quantization, and batching strategies.