Handshake
Senior Software Engineer, LLM Platform
$176K–$220K
San Francisco, CA · On-site · Senior · Full-time · Posted Jul 10, 2026
About the job
Build and operate shared LLM platform infrastructure, including gateways, provider integrations, access controls, observability, and inference services.
What you'll do
- Build and operate LiteLLM gateways and shared LLM clients; manage provider and model onboarding, routing, failover, rate limits, and capacity.
- Create access, credential, budget, observability, cost-attribution, and reliability capabilities for production LLM traffic.
- Contribute to workflow orchestration, model serving, cloud infrastructure, and developer tooling; participate in on-call support.
What they ask for
- Production software engineering experience in Python, TypeScript, Go, or a similar language.
- Experience building or operating high-throughput API gateways, proxies, or multi-tenant platform services.
- Hands-on experience with Kubernetes, Terraform, CI/CD, and production service ownership.
- Practical experience with LLM provider APIs and production concerns such as streaming, retries, timeouts, cancellation, and rate limits.
Nice to have
- Experience with LiteLLM, Portkey, or a comparable multi-provider AI gateway.
- Experience with GPU-backed serving or durable execution systems such as vLLM, Modal, Ray, Triton, or Temporal.
Pay
$176K – $220K USD base salary
Tools
Python, TypeScript, Go, LiteLLM, Kubernetes, Terraform, BigQuery, Airflow
AI focus
Build and operate LLM gateways and inference infrastructure, supporting hosted and self-hosted models and production AI workflows.
Model-written summary · the original posting is authoritative