Pluralis Research
Research Scientist - Pre-training
San Francisco +1 · Remote · Full-time · Posted Oct 14, 2026
- Location
- San Francisco
Australia - Visa sponsorship
- Optional full visa sponsorship and relocation support to Australia or the US.
About the job
Build and optimize distributed pretraining systems to train frontier-scale models across heterogeneous consumer hardware connected over the internet.
What you'll do
- Implement and optimize data, pipeline, and tensor parallel training across heterogeneous GPUs and challenging networks.
- Reduce communication overhead while maintaining model convergence; build fault-tolerant checkpointing, synchronization, and recovery for changing participants.
- Instrument runs to monitor throughput, bottlenecks, and model quality across devices.
What they ask for
- Hands-on distributed training across many devices using PyTorch, with FSDP, DeepSpeed, Megatron, or a custom implementation.
- Understanding of data, tensor, and pipeline parallelism.
- Production-quality Python skills, including concurrency, failure handling, and profiling.
- Evidence of execution through shipped systems, research code, open-source work, or serious personal projects.
Nice to have
- Experience training or serving large language models such as Nemotron, Qwen, or OLMo.
- Experience with P2P networking and NAT traversal, post-training and RL, inference and serving systems, or AI labs.
Tools
Python, PyTorch, FSDP, DeepSpeed, Megatron, Nemotron, Qwen, OLMo
AI focus
Develop distributed pretraining systems for Protocol Learning, including training frontier-scale models across decentralized, heterogeneous devices.
Model-written summary · the original posting is authoritative