NVIDIA
Principal Software Engineer, At-Scale Reliability and Fleet Intelligence — CSP Engagements
$272K–$431K
US, CA, Santa Clara · Principal · Full-time · Posted Oct 13, 2026
- Experience
- 15+ years in systems software at datacenter scale, or reliability engineering focused on at-scale challenges.
- Education
- BS or MS in Computer Science, Electrical Engineering, Statistics, or related field (or equivalent experience)
About the job
Lead fleet-scale reliability work with hyperscale customers, analyze reliability data, and drive improvements to NVIDIA platforms and monitoring systems.
What you'll do
- Coordinate reliability work streams with CSP teams, including MTBI methodology, health monitoring, and launch readiness.
- Analyze fleet failure patterns and telemetry, then champion improvements across firmware, driver, hardware, and quality teams.
- Define burn-in and certification criteria and develop predictive failure models validated in customer environments.
What they ask for
- Deep expertise in multi-NUMA, rack-scale system software and firmware, and statistical failure analysis.
- Experience with fleet-level telemetry and observability systems, including time-series databases, anomaly detection, health scoring, or event correlation.
- Understanding of hardware failure modes in large-scale GPU or accelerator deployments and experience with burn-in, stress testing, or certification frameworks.
- Strong communication and demonstrated success driving cross-functional improvements across hardware, firmware, and software teams.
Nice to have
- Fleet reliability experience at a hyperscaler; familiarity with NVIDIA GPU error taxonomy.
- Experience building health-scoring or predictive failure models for accelerator or HPC infrastructure.
Pay
272,000 USD - 431,250 USD (base salary)
Tools
MTBI, MTBF, Pareto, survival analysis, Weibull, time-series databases, Xid errors, NVLink error counters
AI focus
Develop predictive failure models using fleet telemetry and validate their effectiveness in customer environments.
Model-written summary · the original posting is authoritative