Helix AI Engineer, Training Performance

What you'll do

• Optimize training performance and distributed training configurations for models with 100B+ parameters across 100k+ GPUs.
• Develop custom kernels, performance monitoring tools, and automated kernel-generation and benchmarking systems.
• Evaluate accelerators and collaborate on hardware, model, and training-recipe choices.

What they ask for

• Bachelor's or Master's degree in Computer Science, Computer/Electrical Engineering, or a related field.
• 3+ years in AI performance engineering, including significant time leading large-scale performance improvement projects.
• Deep understanding of GPU architecture and performance profiling; proficiency with profiling tools.
• Strong Python and CUDA/C++ skills, with knowledge of collective communication and modern networking concepts.

Nice to have

• Experience with heterogeneous or multi-datacenter training setups and cross-cluster orchestration.
• Open-source ML systems contributions or exposure to non-NVIDIA accelerators.

Pay

$200,000 to $400,000 per year (USD, base salary)

Tools

Triton, CUDA, Gluon, Nsight Systems/Compute, PyTorch Profiler, HTA, NCCL, RDMA

AI focus

Improves performance and hardware utilization for large-scale model training; co-designs model architectures and training recipes, and builds agentic systems to generate, benchmark, and iterate on custom kernels.