OpenAI
Software Engineer, Hardware Health
$250K–$445K
San Francisco · Full-time · Posted May 11, 2026
- Experience
- 7+ years of industry experience in software or infrastructure engineering.
About the job
Build and operate infrastructure that detects, remediates, and verifies hardware problems across large-scale compute clusters, keeping resources available for training and inference.
What you'll do
- Define health signals and build low-latency checks that detect, remediate, and verify failures across compute infrastructure.
- Investigate system and hardware failures; own node lifecycle workflows from drain and quarantine through repair and return to service.
- Develop automation and operational tooling for global cluster management, partnering with workload, reliability, and provider teams.
What they ask for
- Strong proficiency with Python and shell scripting.
- Experience building large-scale distributed systems or infrastructure platforms.
- Comfort analyzing operational data with SQL, PromQL, or similar tooling.
- Experience building reproducible analyses and operational tooling.
Nice to have
- Experience with low-level hardware systems, Linux tooling, or large-scale GPU/accelerator clusters.
- Experience in network operations, observability, systems telemetry, automated remediation, or fleet lifecycle management.
Pay
$250K – $445K USD base salary
Tools
Python, SQL, PromQL, PCIe, InfiniBand, RoCE, Linux
AI focus
Supports the reliable operation of compute used for frontier model training and inference; the role does not involve developing AI models.
Model-written summary · the original posting is authoritative