Add an LMCache L2 adapter and correct the KV and pipelined loader claims

HyperCache has no real LMCache integration, and some docs claim more than the code does. The fixes are on branch feat/lmcache-l2-adapter, awaiting review. They are not merged.

Problems found

KV blocks get uploaded to the backend. handle_stage_write ignores auto_flush_s3=False, and every commit starts write-back unless it is write-through. HyperCacheKVStore therefore tries to upload prompt-derived KV data. If the upload fails, the block stays SYNC_FAILED, dirty and impossible to evict.
locate returns HIT for staged objects. The layout is published before the data is written, so HyperCacheKVStore could return garbage during a write or after a crash.
LMCache and Dynamo KVBM were described as ready. kv_store.py and the README said "plug-and-play" and "LMCache / Dynamo KVBM Ready", but neither integration existed.
The pipelined loader claim is wrong. The README says Layer 0 loads in milliseconds so compute "starts immediately", and pipeline.py says "sub-100ms perceived cold starts". The first token needs every layer, so the saving is at most min(T_io, T_compute). The loader also returns bytes only and runs no model.
The README's pipelined loader example doesn't match the API. prefetch_depth and stream_layers(layer_uris) do not exist.
DirectBlockWriter ignores short writes.

Measurements

Small test server: RTX 4060 Ti 16 GB, Ryzen 5 5600GT, Intel Gen3 x2 NVMe (about 1.4 GB/s direct read). Treat these as behaviour and relative cost, not production throughput.

Pipelined loader, Qwen2.5-7B-Instruct, 15.2 GB, median of 3 runs:

Prompt tokens	First layer	Eager TTFT	Pipelined TTFT	Change
5	880 ms	15.14 s	15.66 s	3.4% slower
2048	878 ms	15.88 s	16.07 s	1.2% slower
4096	881 ms	16.67 s	16.49 s	1.1% faster

The cause of the slowdown on short prompts was not profiled. Much faster storage could change these percentages, and that was not tested.

vLLM 0.31.0, LMCache 0.5.5 and the new adapter, prompt of about 9,500 tokens:

Request	TTFT	Served from
Cold	3.55 s	Computed
Repeat	0.20 s	L1
After L1 was filled	0.68 s	NVMe (37 of 37 chunks in 459 ms, one run)

Output text was identical across all three. A raw-device speed comparison against LMCache's adapters is in progress and will be added as a comment.

What the branch changes

Adapter: adds hypercache.integrations.lmcache_l2 as a plugin L2 adapter, with an optional lmcache extra.
Daemon: adds a LOCAL_ONLY status (never flushed, stays evictable), optional local_only and chunk_checksums on commit_write, and sync_status on locate HITs. All are backward compatible on the wire.
Safety: the adapter treats any doubt as a miss, and verifies a CRC32 for every chunk on load. Its namespace setting is required, because LMCache's key has no model revision. It refuses to store against an older daemon before writing any data, so nothing can be uploaded by accident.
Client: HyperCacheKVStore commits local-only unless write_through is set, and treats staged blocks as misses. DirectBlockWriter retries short writes.
Docs: corrects the claims above and the broken example.

Verification

69 Python tests pass with 1 skipped on macOS, including 35 new ones. All 79 Rust library tests pass locally, including 2 new ones.
scripts/e2e/lmcache_l2_e2e.py passes 18 of 18 on LMCache 0.5.5 and 0.5.6 with the real daemon on a raw device, including a flipped byte on disk.
verify_release.sh --quick passes on macOS, with YAML skipped because PyYAML is not installed.
verify_release.sh --stages linux passes in a clean container. The image stage also passed; the k8s stage was skipped.

Still open

Flaky test: test_concurrent_locates_trigger_single_ingest failed 11 of 60 runs under CPU load, and once in the harness. The final check on engine.inflight appears to race with the ingest leader's cleanup. The branch does not touch that file.
Pressure eviction frees extents with no grace period. A reader holding an old layout could read reused bytes. The adapter's CRC check turns that into a miss, but the other readers need review.
Crashed writers: STAGED counts as dirty, so an object from a crashed writer can't be evicted until something rewrites or removes it.
socket_path in the config file: the daemon ignored it in testing and used the default path unless HYPERCACHE_SOCKET_PATH was set.
Not supported by the adapter: list_l2_keys and IsolatedLRU quotas.
Later LMCache versions: get_memory_tags becomes abstract and ObjectKey changes on the dev branch, so the adapter pins <0.6.
LMCache adapter docs: document how to install, configure and use the adapter, and what it does not support. The README has a short section only. Cover GETTING_STARTED.md, client-py/README.md, ARCHITECTURE.md and the hypercache-web docs.
Website docs: hypercache-web (python-sdk.md, architecture.md) repeats the same claims.

Raw-device comparison

All rows use the same drive (Intel SSDPEKNU512GZ, Gen3 x2, about 1.6 GB/s read), wiped before every run. 14 MiB objects, 8 keys per submit, 4 in flight, 20 rounds. Median of 3 runs. This shows relative cost on one small drive, not production throughput.

Adapter	Mode	Workers	Store MB/s	Load MB/s
LMCache `raw_block`	Raw	8	1366	1534
LMCache `raw_block`	Raw	Default	1327	1534
LMCache `fs_native`, O_DIRECT	ext4	Default	1327	1513
HyperCache	Filesystem arena	8	1064	1500
HyperCache	Raw	2	906	1423
HyperCache	Raw	4	1098	1390
HyperCache	Raw	8	1127	1144

The arena row and its fs_native comparison used LMCache 0.5.5 (fs_native at 8 workers: 1343 and 1514). The other rows used 0.5.6. raw_block with io_uring could not start in this container, so it was not measured.

HyperCache does not beat LMCache's built-in adapters on this drive. Raw mode with 4 workers is about 9% slower on load and 20% slower on store than raw_block. HyperCache does verify a CRC32 per chunk, costing about 3% of read throughput in a direct probe. raw_block's source shows CRC checks only on metadata checkpoints.
HyperCache raw mode is slower to load than its own arena mode.
Parallel writes slow later raw reads. With no LMCache involved, data read back at about 1600 MB/s after serial writes, 1250 after 8-way writes and 1145 after 16-way writes. Read parallelism made almost no difference. raw_block writes in parallel and still loads at 1534 MB/s, so this looks specific to HyperCache's raw write path. Writing chunks through the page cache from a shared pool may scatter them inside the drive, but that is unverified.

The adapter's num_workers default is now 4 because of this.

To investigate: write each object's chunks in order, or with O_DIRECT, and re-measure raw store and load speed.

Update: raw write path fixed on the branch

My earlier comment guessed that out-of-order chunks were the cause of slow raw reads. That guess was wrong. Testing each write method separately showed that writing through the page cache is what lowers later read speed. Writing chunks in order made no improvement.

At 8 objects in parallel, on a freshly wiped drive, with no LMCache involved:

Write method	Write MB/s	Later read MB/s	Commit median
Page cache, today's path	1189	1214	77 ms
Page cache, chunks in order	1089	1175	81 ms
O_DIRECT, parallel chunks	1363	1604	20 ms
O_DIRECT, chunks in order	1394	1611	13 ms

Why page-cache writeback has this effect on this drive was not investigated, so other drives may behave differently.

The change: DirectBlockWriter gains an opt-in direct_on_block_devices mode, used only by the LMCache adapter and only on raw block devices. Partial last blocks, refused writes and devices without O_DIRECT go through the page cache. Unaligned buffers go through an aligned copy. The default path, the disk format and the protocol are unchanged.

Results through the adapter, LMCache 0.5.6, same drive, median of 3 runs in shuffled order:

Adapter	Workers	Store MB/s	Load MB/s
LMCache `raw_block`	8	1367	1538
HyperCache raw, before	4	1098	1390
HyperCache raw, before	8	1127	1144
HyperCache raw, now	4 (default)	1313	1532
HyperCache raw, now	8	1314	1532

HyperCache now loads within 0.4% of raw_block and stores about 4% slower, while verifying a CRC32 per chunk. It does not beat raw_block, and these loads are near this drive's limit of about 1.6 GB/s.
Unaligned L1 buffers cost about 18% on stores (1071 MB/s). LMCache's default L1 alignment is 4096, so that should not be the usual case.
raw_block with io_uring could not start in this container and was not measured.
vLLM through the adapter: 0.62 s TTFT from NVMe (one run; 37 of 37 chunks in 402 ms), against 3.56 s cold, with identical output.

Verification: the adapter's 18 end-to-end checks pass on LMCache 0.5.5 and 0.5.6 on a raw device, including rewriting the same extents alternately through the page cache and O_DIRECT. verify_release.sh --stages linux passes in a clean container.

The earlier to-do on writing chunks in order is closed by these results. Still open: daemon model ingest and other DirectBlockWriter callers still write through the page cache, and were not tested with O_DIRECT.

Closing in favour of #4, which tracks the Rust connector that replaced the Python adapter described here.

Where each part stands:

Problems 1 to 6: fixed on feat/lmcache-rust-connector, not on main yet. Get the Rust LMCache connector to main #4 tracks merging them.
Python adapter: removed. It remains in git history on feat/lmcache-l2-adapter at 5108dcc.
Still-open items: moved to Get the Rust LMCache connector to main #4, along with the remaining docs work.
Raw-device speed comparison: finished. See lmcache_rust_connector_nvme.md on the branch.
Related fix now on main: fix(striper): start each object on its own drive #3, where each object now starts on its own drive.

Add LMCache L2 adapter and correct KV and pipelined loader claims

Navigation Menu

Repository navigation

Add an LMCache L2 adapter and correct the KV and pipelined loader claims #2

Problems found

Measurements

What the branch changes

Verification

Still open

Raw-device comparison

Update: raw write path fixed on the branch

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add an LMCache L2 adapter and correct the KV and pipelined loader claims #2

Description

Problems found

Measurements

What the branch changes

Verification

Still open

Activity

neo-th commented on Nov 2, 2026

Raw-device comparison

neo-th commented on Nov 2, 2026

Update: raw write path fixed on the branch

neo-th commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions