OpenAI
Training Performance Engineer
$295K–$500K
San Francisco · Hybrid · Full-time · Posted Oct 16, 2025
About the job
Improves the performance and reliability of OpenAI’s distributed training stack so large models can train with higher throughput and the same compute budget.
What you'll do
- Profile training runs and address compute, communication, and storage bottlenecks to improve throughput and uptime.
- Optimize GPU utilization, kernels, scheduling, collective communication, and model graph transforms; build cluster monitoring tools.
- Partner with researchers and systems teams to help model architectures and large training jobs scale efficiently and reliably.
What they ask for
- Strong programming skills in Python and C++.
- Experience running distributed training jobs on multi-GPU systems or HPC clusters.
- Exposure to PyTorch, JAX, or TensorFlow and understanding of large-scale training loops.
- Ability to debug distributed systems, measure efficiency, and turn profiling data into engineering improvements.
Nice to have
- Rust or CUDA; familiarity with NCCL, MPI, or UCX.
- Experience with large-scale data loading or checkpointing, training runtime, distributed scheduling, or ML compiler optimization.
Pay
$295K – $500K USD base salary
Tools
Python, C++, Rust, CUDA, PyTorch, JAX, TensorFlow, NCCL
AI focus
Improves performance and scalability of distributed model training, including pre-training for new model architectures.
Model-written summary · the original posting is authoritative