feat: add a Rust LMCache L2 connector

.github/workflows/ci.yml

@@ -30,7 +30,9 @@ jobs:
       - name: Cache Cargo registry and target
         uses: Swatinem/rust-cache@v2
         with:
-          workspaces: "control-plane -> target"
+          workspaces: |
+            control-plane -> target
+            lmcache-connector -> target
 
       - name: Set up Python
         uses: actions/setup-python@v5
@@ -43,6 +45,16 @@ jobs:
           python -m pip install "numpy>=1.22" "fsspec>=2023.1.0" pytest boto3 safetensors pyyaml build
           sudo apt-get install -y -qq lsof
 
+      # In this job, not its own, so the check that main requires covers the connector too.
+      # The tests need no Python or LMCache. The Python bindings are only linted here.
+      - name: LMCache connector
+        working-directory: lmcache-connector
+        run: |
+          cargo fmt --check
+          cargo clippy --locked --all-targets -- -D warnings
+          cargo clippy --locked --features extension-module -- -D warnings
+          cargo test --locked
+
       # Runs every stage except `linux` (the runner already is a clean Linux host) and `k8s`
       # (needs minikube). The s3 stage starts a throwaway versitygw container, and the image
       # stage builds the release image and tests host access to the containerized daemon.

AGENTS.md

@@ -93,6 +93,7 @@ hypercache-poc/
 │   └── tests/
 │       └── s3_live.rs          # #[ignore]d live S3 tests (run by scripts/verify_release.sh)
 │
+├── lmcache-connector/          # LMCache native L2 connector in Rust (PyO3, built with maturin); separate from the daemon
 ├── scripts/
 │   ├── verify_release.sh       # Release verification harness (static, rust, python, e2e, s3, linux, image)
 │   ├── e2e/hypercache_e2e.py   # End-to-end daemon / fleet / S3 scenarios against the real binaries
@@ -110,14 +111,15 @@ hypercache-poc/
     │   ├── datapath.py         # Direct Block DMA / GDS direct I/O reader
     │   ├── fsspec_fs.py        # AbstractFileSystem implementation for torch.load/save
     │   ├── kv_store.py         # Content-addressed KV-Cache Prefix Block Store
-    │   ├── pipeline.py         # Double-buffered pipelined GPU tensor loader
+    │   ├── pipeline.py         # In-order byte-range reader with background prefetch; returns bytes, runs no model
     │   └── integrations/
     │       ├── safetensors_loader.py # Zero-copy Safetensors & vLLM weight loader
     │       └── dcp.py          # PyTorch Distributed Checkpoint (DCP) StorageReader/Writer
     └── tests/
         ├── test_client.py      # Unit tests for UDS protocol and client metadata parsing
         ├── test_fsspec.py      # Unit tests for fsspec file operations and streaming writes
         ├── test_integrations_and_kv.py # Unit tests for Safetensors/vLLM, DCP, and KV-Cache store
+        ├── test_direct_writer.py       # DirectBlockWriter O_DIRECT mode: alignment, partial blocks, fallbacks
         ├── test_pipelined_and_numa.py  # Unit tests for pipelined DMA and NUMA affinity
         └── test_robustness.py  # Thread/fork safety, framing, O_DIRECT bounce buffers, CRC verification
 ```

ARCHITECTURE.md

@@ -391,6 +391,17 @@ dataset = ShardedTokenDataset(["hypercache://datasets/tokens/shard_000.bin", ...
 loader = DataLoader(dataset, batch_size=16, num_workers=4)
 ```
 
+### 7.5. LMCache KV-Cache Connector
+
+`lmcache-connector/` is an L2 connector for LMCache's multiprocess mode, written in Rust and loaded through LMCache's `native_plugin` type. It's a separate package from the Python client.
+
+- **Control path:** each worker thread keeps a connection to the daemon's socket for `locate`, `stage_write` and `commit_write`. Commits are `LOCAL_ONLY`, so KV data is never flushed to a backend and stays evictable.
+- **Data path:** workers read and write chunks directly on the devices the layout names, with `O_DIRECT` into LMCache's page-aligned L1 buffers, and verify a CRC32 per chunk on load.
+- **Failure rule:** any doubt is a miss. A block that fails its checksum is evicted. If the daemon restarts, workers reconnect, and a request the daemon may have received is never resent.
+- **Concurrency:** a worker handles one object at a time and reads its chunks in order, so `num_workers` sets the queue depth.
+
+See [`lmcache-connector/README.md`](lmcache-connector/README.md).
+
 ---
 
 ## 8. CLI Command Reference & Operator Guide

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 ## Unreleased
 
+### Added
+- `lmcache-connector/`: an L2 connector for LMCache multiprocess mode, written in Rust and loaded
+  through LMCache's `native_plugin` type. Built separately with maturin. Tested on LMCache 0.5.5 and
+  0.5.6 and vLLM 0.31.0 and 0.33.0.
+- `commit_write` accepts `local_only` and `chunk_checksums`. A `LOCAL_ONLY` object is never flushed
+  to a backend and stays evictable. Both fields are optional on the wire; older clients are unaffected.
+- `locate` reports `sync_status` for HITs, so a reader can refuse an object that is still staged.
+- `DirectBlockWriter(direct_on_block_devices=True)` writes to raw block devices with O_DIRECT, falling
+  back to the page cache for anything it can't take. Off by default.
+- `benchmarks/pipelined_ttft.py`, `benchmarks/write_concurrency_probe.py`, `benchmarks/raw_write_probe.py` and
+  `benchmarks/results/lmcache_and_ttft.md`.
+- `benchmarks/lmcache_connector_direct.py`, `benchmarks/results/lmcache_rust_connector.md`,
+  `benchmarks/results/lmcache_rust_connector_xeon.md` and
+  `benchmarks/results/lmcache_rust_connector_nvme.md`.
+
 ### Fixes
 - Each object now starts on its own drive, chosen from a hash of its URI. Previously every object
   started on the first drive. Objects that don't fill a whole stripe left the last drives underused,
@@ -11,6 +26,18 @@
   objects load at 24884 MB/s (was 20740) and store at 18549 MB/s (was 16129).
 - Upgrading needs no migration. Objects already stored keep their placement and stay readable, and
   objects stored after the upgrade stay readable if you roll back. Existing objects aren't rebalanced.
+- `HyperCacheKVStore` commits are now `LOCAL_ONLY` unless `write_through` is set. Previously every
+  commit started a backend upload of prompt-derived data, ignoring `auto_flush_s3=False`, and a failed
+  upload left the block dirty and impossible to evict. Reads now treat staged blocks as misses.
+- `DirectBlockWriter` retries short writes instead of ignoring them, no longer copies each chunk, and
+  no longer leaks a file descriptor when two threads open the same device.
+
+### Docs
+- Corrected the pipelined loader claims. It does not shorten time to first token by itself: on a
+  15 GB model the change was between -3.4% and +1.1%, and the first layer took 880 ms, not milliseconds.
+- Fixed the pipelined loader README example, which did not match the API.
+- Removed the claims of LMCache and Dynamo KVBM readiness. LMCache now has a real adapter; Dynamo
+  KVBM has no integration.
 
 ## 0.9.3
 

GETTING_STARTED.md

@@ -309,3 +309,35 @@ scrape_configs:
 4. **Proxies and Sidecars**: If the controller sits behind a local proxy (Istio/Envoy sidecar, nginx, `kubectl port-forward`), set `allow_local_admin_bypass = false` in `hypercache-controller.toml`. Otherwise forwarded remote calls arrive from `127.0.0.1` and skip the token check.
 5. **Restrict Filesystem Access**: `file://` and `nfs://` URIs may only reach `nfs_mount_path` and the directories listed in `nfs_allowed_paths`. Add dataset or checkpoint directories there explicitly instead of widening `nfs_mount_path` to `/`.
 6. **Verify Before Rollout**: Run `scripts/verify_release.sh` on the build you deploy. It exercises the daemon, the fleet controller and a live S3 endpoint end to end.
+
+---
+
+## 9. KV-Cache Storage for LMCache
+
+The connector in `lmcache-connector/` lets an LMCache server in multiprocess mode keep KV-cache chunks on HyperCache's NVMe tier. It's tested with LMCache 0.5.5 and 0.5.6.
+
+**1. Build and install it.** It isn't published as a package yet, so this needs a Rust toolchain and maturin. Use the same Python that runs LMCache:
+
+```bash
+cd lmcache-connector
+maturin build --release -i python3.12
+pip install target/wheels/hypercache_lmcache_native-*.whl
+```
+
+**2. Start the daemon** on the node, as in section 3.
+
+**3. Start LMCache with the connector:**
+
+```bash
+lmcache server --l1-size-gb 60 --eviction-policy LRU --l1-align-bytes 4096 --l2-adapter '{
+  "type": "native_plugin",
+  "module_path": "hypercache_lmcache_native",
+  "class_name": "HyperCacheNativeConnector",
+  "adapter_params": {"namespace": "my-model-rev-abc123", "tier": "nvme", "num_workers": 32},
+  "max_capacity_gb": 0
+}'
+```
+
+Change `namespace` whenever the model weights, dtype or engine settings change, because LMCache's key has no model revision. Keep `max_capacity_gb` at 0, since the daemon owns capacity. Use `num_workers` 8 on one drive and 32 on four.
+
+The settings, behaviour and limits are in [`lmcache-connector/README.md`](lmcache-connector/README.md), and measurements are in [`benchmarks/results/lmcache_rust_connector_nvme.md`](benchmarks/results/lmcache_rust_connector_nvme.md).

README.md

@@ -107,7 +107,7 @@ HyperCache is designed for both **AI Inference** and **Distributed AI Training**
 
 ### 1. Fast LLM Inference Cold Starts & Serverless GPU Scaling
 * **Pre-Warm Models Fleet-Wide**: Use `./hypercache-admin preload` to stage base models onto local NVMe or RAM across your cluster ahead of traffic spikes.
-* **Layer-by-Layer Pipelined Loading (`HyperCachePipelinedLoader`)**: Instead of waiting for 100% of a model to load before starting inference, HyperCache loads Layer 0 in milliseconds so GPU forward-pass compute starts immediately while background DMA threads prefetch Layers $1, 2, \dots, N$ ahead of execution.
+* **Layer-by-Layer Pipelined Loading (`HyperCachePipelinedLoader`)**: Reads a model's byte ranges in order and prefetches the next ones in the background, so an engine that can run layers as their weights arrive can overlap compute with I/O. It does not shorten time to first token by itself: the first token needs every layer, so the saving is at most `min(T_io, T_compute)`. In [our test](benchmarks/results/lmcache_and_ttft.md) on a 15 GB model, time to first token moved between -3.4% and +1.1%.
 * **Zero-Copy Multi-Tenant Model Sharing (`MAP_SHARED`)**: If you run multiple inference worker processes or containers on the same machine, they map the exact same physical RAM pages in Tier 1 shared memory with **0 bytes of memory duplication**.
 
 ### 2. Sub-Millisecond Multi-LoRA Adapter Hot-Swapping
@@ -124,7 +124,7 @@ HyperCache is designed for both **AI Inference** and **Distributed AI Training**
 * Each node caches only its assigned $1/N$ slice of the dataset and streams batches into local PyTorch `DataLoader` workers at local NVMe line rate with zero network traffic. Because it uses consistent hashing, adding or removing a node only remaps $1/N$ of the shards instead of invalidating the entire cluster cache.
 
 ### 5. LLM KV-Cache Prefix Block Offloading (`HyperCacheKVStore`)
-* Long-context LLM serving generates massive KV-cache tensors. Rather than recomputing shared prompt prefixes when GPU VRAM fills up, inference engines (and frameworks like LMCache or Dynamo KVBM) can use `HyperCacheKVStore` to offload content-addressed KV-cache blocks into Tier 1 Host RAM with automatic spillover to Tier 2 NVMe.
+* Long-context LLM serving generates massive KV-cache tensors. Rather than recomputing shared prompt prefixes when GPU VRAM fills up, LMCache can keep them on HyperCache through [`lmcache-connector/`](lmcache-connector/README.md), and other code can use `HyperCacheKVStore` directly. Blocks stay on the local host and remain evictable. There is no NVIDIA Dynamo KVBM integration.
 
 ### 6. Peer-to-Peer (P2P) Fleet Distribution
 * When preloading a large model across a multi-node cluster, you do not want every node pulling the full file from S3 simultaneously.
@@ -679,21 +679,42 @@ state_dict = reader.load_rank_shard()
 
 ### 5. Layer-by-Layer Pipelined Model Loader (Compute / I/O Overlap)
 
-Start forward-pass compute on Layer 0 immediately while subsequent layers prefetch in the background:
+Read a model in order while the next byte ranges prefetch in the background:
 
 ```python
 from hypercache.pipeline import HyperCachePipelinedLoader
 
-loader = HyperCachePipelinedLoader(prefetch_depth=2)
+loader = HyperCachePipelinedLoader("hypercache://models/model.bin", num_layers=30, prefetch_ahead=2)
 
-# Layer 0 arrives immediately; Layers 1..N prefetch in background threads during GPU compute
-for layer_idx, layer_bytes in loader.stream_layers(layer_uris):
-    run_layer_forward_pass(layer_idx, layer_bytes)
+for layer, layer_bytes in loader.stream_layers():
+    run_layer_forward_pass(layer.layer_idx, layer_bytes)
 ```
 
-### 6. KV-Cache Prefix Block Store (LMCache / Dynamo KVBM Ready)
+The loader splits the object by bytes and returns bytes. Your code has to supply true layer boundaries (`loader.layers`), turn bytes into tensors and run the model. The first token still needs every layer, so the saving is at most `min(T_io, T_compute)`. See the [measurements](benchmarks/results/lmcache_and_ttft.md).
 
-Offload and retrieve prompt prefix KV-cache blocks on striped local NVMe (or optional Host RAM):
+### 6. KV-Cache Storage for LMCache
+
+[`lmcache-connector/`](lmcache-connector/README.md) is an L2 connector for LMCache's multiprocess mode, written in Rust. It's tested with LMCache 0.5.5 and 0.5.6 and vLLM 0.31.0 and 0.33.0, and needs a daemon built from this version or later. Build it with maturin, install the wheel where the LMCache server runs, and pass:
+
+```bash
+lmcache server --l1-size-gb 60 --eviction-policy LRU --l2-adapter '{
+  "type": "native_plugin",
+  "module_path": "hypercache_lmcache_native",
+  "class_name": "HyperCacheNativeConnector",
+  "adapter_params": {"namespace": "my-model-rev-abc123", "tier": "nvme"},
+  "max_capacity_gb": 0
+}'
+```
+
+- **`namespace` is required.** LMCache's key has no model revision or dtype, so change this value whenever the weights or dtype change. Otherwise a new model could read the old model's KV data.
+- **Blocks stay local.** They are never flushed to a backend, because they derive from user prompts and can be recomputed. They stay evictable, and the daemon manages capacity.
+- **Doubt means a miss.** Every load verifies a CRC32 per chunk. A failed check reports a miss and removes the block.
+- **Not supported:** `list_l2_keys` and LMCache's IsolatedLRU quotas.
+- **Size L1 for loads.** Loads go through L1, and LMCache only evicts it down to 80%. If the load doesn't fit, vLLM recomputes.
+
+On 4 Gen4 NVMe drives it loads 14 MiB objects at 24.9 GB/s and stores them at 18.5 GB/s, while verifying a CRC32 on every chunk. That's 90% of the drives' raw read speed. LMCache's `raw_block` adapter uses one drive, where the two are about even on throughput, and its lookups are about 3 times as fast. See the [NVMe results](benchmarks/results/lmcache_rust_connector_nvme.md).
+
+For code that manages its own KV blocks, use `HyperCacheKVStore`:
 
 ```python
 from hypercache import HyperCacheKVStore
@@ -706,7 +727,7 @@ block_key = kv_store.compute_prefix_hash(
 )
 
 # Store KV-cache block into local striped NVMe
-kv_store.put_kv_block(block_key, kv_tensor, tier="nvme", pinned=True)
+kv_store.put_kv_block(block_key, kv_tensor, tier="nvme")
 
 # Retrieve KV-cache block directly into pre-allocated GPU tensor
 kv_store.get_kv_block(block_key, out_tensor=gpu_kv_buffer)

benchmarks/lmcache_connector_direct.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+"""
+Measures the Rust LMCache connector with LMCache out of the path, to find what limits it.
+
+Calls the connector the way LMCache's wrapper does, with page-aligned buffers. Needs a running
+daemon, and the hypercache_lmcache_native package.
+
+    python3 benchmarks/lmcache_connector_direct.py --socket /run/hc/hc.sock --tier ram
+"""
+
+import argparse
+import json
+import mmap
+import os
+import select
+import time
+
+from hypercache_lmcache_native import HyperCacheNativeConnector
+
+OBJECT_SIZE = 14 * 1024 * 1024
+OBJECTS = 64
+BATCH = 8
+MIB = 1024 * 1024
+
+
+def make_buffers():
+    seed = os.urandom(MIB)
+    buffers = []
+    for _ in range(OBJECTS):
+        view = memoryview(mmap.mmap(-1, OBJECT_SIZE))
+        for offset in range(0, OBJECT_SIZE, MIB):
+            view[offset : offset + MIB] = seed
+        buffers.append(view)
+    return buffers
+
+
+def wait(connector, future_ids):
+    pending = set(future_ids)
+    all_ok = True
+    while pending:
+        select.select([connector.event_fd()], [], [], 1)
+        for future_id, ok, _error, _results in connector.drain_completions():
+            all_ok = all_ok and ok
+            pending.discard(future_id)
+    return all_ok
+
+
+def timed(connector, submit, keys, buffers, in_flight_keys):
+    start = time.perf_counter()
+    future_ids = []
+    all_ok = True
+    for i in range(0, OBJECTS, BATCH):
+        if len(future_ids) * BATCH >= in_flight_keys:
+            all_ok = wait(connector, future_ids) and all_ok
+            future_ids = []
+        future_ids.append(submit(keys[i : i + BATCH], buffers[i : i + BATCH]))
+    all_ok = wait(connector, future_ids) and all_ok
+    return OBJECTS * OBJECT_SIZE / 1e6 / (time.perf_counter() - start), all_ok
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--socket", required=True)
+    parser.add_argument("--tier", default="nvme", choices=["nvme", "ram"])
+    args = parser.parse_args()
+
+    buffers = make_buffers()
+    keys = [f"m@00000000@0@{i:04x}" for i in range(OBJECTS)]
+    for workers, in_flight_keys in [(4, 64), (8, 64), (16, 64), (32, 64), (16, 8)]:
+        for repeat in range(2):
+            namespace = f"direct-{os.getpid()}-{workers}-{in_flight_keys}-{repeat}"
+            connector = HyperCacheNativeConnector(namespace, args.socket, args.tier, workers)
+            store, store_ok = timed(connector, connector.submit_batch_set, keys, buffers, in_flight_keys)
+            load, load_ok = timed(connector, connector.submit_batch_get, keys, buffers, in_flight_keys)
+            print(
+                json.dumps(
+                    {
+                        "workers": workers,
+                        "in_flight_keys": in_flight_keys,
+                        "store_mbps": round(store),
+                        "load_mbps": round(load),
+                        "ok": store_ok and load_ok,
+                    }
+                ),
+                flush=True,
+            )
+            wait(connector, [connector.submit_batch_delete(keys)])
+            connector.close()
+
+
+if __name__ == "__main__":
+    main()