lmcache

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.

Qwen3.8-27B on 2× RTX 5090 · 1.49M KV · code/prose t/s: c1 264/216, c16 2,649/2,240 · NVFP4 + vLLM + MTP

PCIe multi-GPU deployment & inference optimization for GLM-5.3-Flash W4A16 / DeepSeek-V4-Flash / V4.1-Flash on A800/A40: CUDA graphs, MTP speculative decoding, LMCache KV offload, NCCL tuning, concurrent benchmarks. 多卡部署与推理优化实测。

Upstream-maintained home for vendor L2 adapter plugins for LMCache

High-performance host-local NVMe caching engine for AI clusters.

Multimodal LLM inference gateway with KV-cache-aware routing and LMCache offload. OpenAI-compatible, benchmarked on GPUs.

Zero-Trust Cold Pages (ZTCP): Single-pass fused RoPE-preserving KV-cache quantization & RFC 8439 ChaCha20 + SipHash-2-4-128 authenticated encryption (26.65 GB/s SIMD) for distributed RDMA LLM serving (SGLang, vLLM/LMCache, Mooncake)

Reproducible LLM KV-cache tiered-storage benchmark on AMD MI308X: LMCache parallel-read patch, load clients, orchestration, measured results

Exploring how to optimize build times for ROCm-based AI frameworks using Buildkit, sccahe and other methods.

KV-cache-aware LLM inference mesh on a single 6 GB GPU. 2× vLLM + LMCache shared KV + prefix-aware router on k3s, observed with Cilium/Hubble eBPF. Every number measured, every breakage documented.

Upstream-maintained home for vendor device plugins for LMCache

Measured TTFT gains from tiered and shared KV cache (vLLM + LMCache) on one NVIDIA L4

Non-prefix KV cache reuse for RAG & agents: CacheBlend-style selective recompute, provenance-keyed cache, reuse gate, and an honest benchmark (MLX + PyTorch/CUDA)

CVE-2026-105192 is a critical (CVSS 9.8) vulnerability in LMCache, an open-source distributed key-value cache for LLM inference engines like vLLM.

CVE-2026-105192 (LMCache): pre-auth pickle deserialization RCE over the unauth ZMQ transport (single frame) — b63 showcase, lab-verified with patched-build negative + reliability/matrix evidence

Collection of benchmarks, benchmark drivers, harnesses, plotting, and various scripts for KV cache offloading in vLLM.

Benchmarking LMCache under simulated RTT