ByteDance
Research Engineer / Scientist - Storage for LLM
San Jose, California · Full-time · Posted Oct 28, 2026
- Education
- PhD in Computer Science, Applied Mathematics, Electrical Engineering, or a related technical field.
About the job
Designs and improves distributed KV-cache infrastructure to reduce latency and compute costs for LLM inference.
What you'll do
- Design and implement distributed KV caching for transformer states across GPUs or nodes, including sharding, eviction, and replication.
- Integrate caching with inference and serving pipelines, and develop consistency and synchronization protocols.
- Monitor performance and improve caching algorithms, including evaluating or extending open-source KV stores.
What they ask for
- PhD in Computer Science, Applied Mathematics, Electrical Engineering, or a related technical field.
- Strong understanding of transformer internals and KV caching in autoregressive decoding.
- Experience with distributed systems, memory management, and low-latency serving.
- Familiarity with high-performance compute environments; proficiency in C++, Rust, Go, or CUDA for systems-level development.
Nice to have
- Prior experience building LLM inference-serving systems.
- Experience with memory hierarchy optimization, GPU-to-GPU communication, or cache-aware scheduling, batching, and prefetching.
Tools
RPC, gRPC, CUDA, NVIDIA GPUs, TensorRT, Triton Inference Server, C++, Rust
AI focus
Builds and optimizes KV caching for LLM inference, including reuse of attention states and prompt embeddings.
Model-written summary · the original posting is authoritative