Liquid AI
Member of Technical Staff - Distributed Training Engineer
San Francisco · Hybrid · Staff · Full-time · Posted Jul 29, 2025
About the job
Build and optimize distributed training infrastructure for Liquid AI’s foundation models, focusing on performance, reliability, and large-scale GPU training.
What you'll do
- Design and build systems for fast, reliable large-scale training on GPU clusters.
- Tune parallelism, sharding, communication, data loading, and checkpointing to improve training efficiency and recovery.
- Create monitoring, profiling, and debugging tools to diagnose training performance and stability issues.
What they ask for
- Hands-on experience building distributed training infrastructure using PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, or Megatron-LM TP/PP.
- Experience diagnosing performance bottlenecks and failures, including profiling, NCCL/collectives issues, hangs, OOMs, and stragglers.
- Understanding of hardware accelerators and networking topologies.
- Experience optimizing data pipelines for ML workloads.
Nice to have
- MoE (Mixture of Experts) training experience, large-scale distributed training (100+ GPUs), or open-source contributions to training infrastructure projects.
Tools
PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP, NCCL
AI focus
Build and optimize infrastructure for training Liquid Foundation Models, including support for evolving architectures and multimodal datasets.
Model-written summary · the original posting is authoritative