General Compute
Inference Engineer
San Francisco +1 · Hybrid · Full-time · Posted Sep 14, 2026
- Location
- San Francisco
NYC - Experience
- 5+ years; no alternatives stated.
About the job
Build and operate the production inference-serving layer for models running on General Compute’s alternative-chip fleet.
What you'll do
- Design and own request routing, batching, scheduling, and autoscaling for model-serving replicas.
- Optimize batching, KV-cache management, and hardware utilization to reduce cost per token.
- Build monitoring, alerting, and failover; respond to reliability issues and collaborate with the compiler and model bring-up team.
What they ask for
- 5+ years building and operating production infrastructure systems, ideally including high-throughput or low-latency serving.
- Production experience with LLM inference serving, such as request batching, KV-cache management, or continuous batching.
- On-call experience and ability to design systems for reliability and failure.
- Strong systems fundamentals in concurrency, networking, and scheduling, including hardware-level performance reasoning.
Nice to have
- Experience serving models on non-NVIDIA accelerators.
- Familiarity with vLLM, TGI, TensorRT-LLM, or SGLang.
Tools
TPU, Trainium/Inferentia, Tenstorrent, Groq, Cerebras, vLLM, TGI, TensorRT-LLM
AI focus
Production LLM inference serving on alternative accelerators, including scheduling, batching, KV-cache management, and autoscaling.
Model-written summary · the original posting is authoritative