Workday
Principal Distributed Systems Engineer - Observability
$187K–$334K
USA, CA, Pleasanton · Flex · Principal · Full-time · Posted Sep 15, 2026
- Experience
- 14+ years in software development engineering; 6+ years focused on complex distributed systems; 8+ years with at least two programming languages.
- Education
- Bachelor’s degree in a relevant field such as Computer Science or Engineering, or equivalent practical experience; a Master’s degree is strongly preferred.
About the job
Own the technical vision and architecture for Workday’s distributed tracing platform, building and operating multi-petabyte observability infrastructure and advancing AI-powered incident analysis.
What you'll do
- Architect and build tracing ingestion, storage, and query systems using the stated data pipeline and observability technologies.
- Lead platform performance, reliability, security, HA/DR, and operational excellence, including on-call participation.
- Set technical direction across teams, mentor engineers, and extend observability to LLM and agentic workflows.
What they ask for
- 14+ years of software development engineering experience.
- 6+ years designing, building, and operating complex distributed systems, including high availability and fault tolerance.
- 8+ years with at least two programming languages such as Java, Python, or Go, including production distributed-systems code.
- Bachelor’s degree in a relevant field or equivalent practical experience.
Nice to have
- Master’s degree in a relevant field.
- Experience with LLM observability tools such as LangSmith and familiarity with LangChain or similar agent orchestration frameworks.
Pay
$222,900 USD - $334,300 USD (base salary); $187,100 USD - $334,300 USD (base salary)
Tools
ClickHouse, Grafana Tempo, Kafka, Spark, Flink, Iceberg, S3, AWS
AI focus
Extend tracing to LLM and agentic workflows and use observability data for automated root-cause analysis, anomaly detection, and AI-driven incident triage.
Model-written summary · the original posting is authoritative