Causal Labs
Member of Technical Staff — Training Infrastructure
San Francisco +1 · On-site · Staff · Full-time · Posted Oct 29, 2025
- Location
- San Francisco
Singapore
About the job
Builds and optimizes distributed training infrastructure for large physics foundation models, enabling efficient, reliable training at scale.
What you'll do
- Design and optimize distributed training systems across thousands of GPUs, including parallelization and numerical precision strategies.
- Profile and debug GPU operations to improve throughput and hardware utilization.
- Build robust checkpointing, fault-tolerance, and reproducibility frameworks, and collaborate with researchers to scale novel architectures.
What they ask for
- Proficiency with distributed training frameworks and techniques for large foundation models.
- Strong grasp of training optimization, including parallelism, memory optimization, mixed precision, and communication overlap.
- Ability to profile and debug performance from framework internals to kernels and collectives.
- Deep understanding of deep learning frameworks and their system architectures.
Nice to have
- Contributions to open-source ML infrastructure, such as PyTorch, Megatron-LM, DeepSpeed, or XLA.
Tools
FSDP, DeepSpeed, Megatron, Pytorch, JAX/XLA, PyTorch, JAX
AI focus
Supports training a Large Physics foundation Model for predicting and controlling physical systems, starting with weather.
Model-written summary · the original posting is authoritative