Pluralis Research
Research Engineer - Pre-training
San Francisco +1 · Remote · Full-time · Posted Aug 31, 2026
- Location
- San Francisco
Australia - Visa sponsorship
- Optional full visa sponsorship and relocation support to Australia or the US.
About the job
Build and optimize distributed pretraining systems to train large models across heterogeneous consumer hardware and unreliable internet connections.
What you'll do
- Implement and optimize data-, pipeline-, and tensor-parallel training across heterogeneous GPUs.
- Reduce communication overhead while preserving model convergence in challenging network environments.
- Build fault-tolerant recovery and monitoring systems for runs with participants joining and leaving.
What they ask for
- Hands-on experience training models across multiple devices with PyTorch and FSDP, DeepSpeed, Megatron, or a custom implementation; understanding of data, tensor, and pipeline parallelism.
- Production-quality Python engineering, including concurrency, failure handling, and profiling.
- Evidence of execution through shipped systems, research code, open-source work, or serious personal projects.
- Professional-level English proficiency, written and spoken.
Nice to have
- Experience training or serving large language models such as Nemotron, Qwen, or OLMo.
- Experience with P2P networking and NAT traversal, post-training and RL, or inference and serving systems.
Tools
PyTorch, FSDP, DeepSpeed, Megatron, Nemotron, Qwen, OLMo
AI focus
Pretraining and serving large models through decentralized Protocol Learning on heterogeneous devices connected over the internet.
Model-written summary · the original posting is authoritative