Repository navigation
Conversation
Adapter: - Add hypercache.integrations.lmcache_l2, a plugin L2 adapter for LMCache multiprocess mode, tested on LMCache 0.5.5 and 0.5.6 and vLLM 0.31.0. - Any doubt about a block is a miss. A required namespace covers the model revision that LMCache's key lacks. Daemon, backward compatible on the wire: - Add a LOCAL_ONLY status: never flushed to a backend, stays evictable. - Add optional local_only and chunk_checksums to commit_write. - Report sync_status on locate HITs. Client: - HyperCacheKVStore commits local-only unless write_through is set, and treats staged blocks as misses. - DirectBlockWriter retries short writes and gains an opt-in O_DIRECT mode for raw block devices. Only the LMCache adapter turns it on. Docs: - Correct the pipelined loader claims using measured time to first token. - Remove the LMCache and Dynamo KVBM readiness claims. - Fix the pipelined loader README example. Results and scripts are in benchmarks/ and scripts/e2e/. Refs #2
Add lmcache-connector/, an L2 connector loaded through LMCache's native_plugin type, and remove hypercache.integrations.lmcache_l2 with its unit tests and the hypercache[lmcache] extra. The Python adapter never shipped in a release and remains at 5108dcc. The connector does its I/O in worker threads without the interpreter lock, and defaults to 32 workers per direction. On a Xeon VM with 4 network disks it loaded at 6.4 GB/s against 2.8 GB/s for Python, and at 39.8 GB/s against 9.4 GB/s from the RAM tier. On one drive the two tie, because the drive sets the limit. The end-to-end script now tests only this connector. Checks that shared blocks with the Python adapter become checks on the URI format. Known open issue: temperature-0 output sometimes differs after a cache hit, including with LMCache L1 only and no HyperCache. See benchmarks/results/lmcache_rust_connector_xeon.md.
Each worker keeps a connection to the daemon. After a restart that connection is dead, so the worker's next request failed with a broken pipe: one failed store or load per worker, with 32 failures at the default worker count. A request that fails while being written can't have reached the daemon, so it now reconnects and sends it again, once. A failure after that point is never resent, because the daemon may already have acted on the request, so a commit can't run twice. Repeated on 4 NVMe drives: no failed first stores after a daemon restart, in all 10 crash and restart tests.
Adds install, configuration and limits to GETTING_STARTED.md, ARCHITECTURE.md, client-py/README.md and the connector's README, and the results from 4 Micron 7450 drives: striping, worker counts, raw_block comparison, mixed reads and writes, failure tests, upgrades and a 30-minute soak.
The page cache check read every chunk through a descriptor for the object's first drive, so it only held for one drive. It failed on 4 drives once objects started on different drives. It had passed before most likely by accident: the test data repeats every 32 bytes, and the old allocator kept drives' offsets in step. Passes 30 of 30 on LMCache 0.5.5 and 0.5.6, on 1 and 4 drives.
Each lookup made one socket round trip per key on 2 workers, which took most of the lookup time. A batch's locate requests now go out in one write, up to 64 keys or 64 KiB, and the replies are read in order. The daemon and its protocol don't change, so this works with existing daemons. At 64 in flight through LMCache on a Ryzen 5 5600, hits went from about 12.0 to 3.7 us per key and misses from about 9.8 to 2.4. raw_block is about 4.3 and 3.9 on the same CPU. If a batch fails part-way, no key in it is reported as found. A timeout or reset drops the connection, so a late reply can't be read as the answer to a later lookup.
…kups Measured on a Ryzen 5 5600: lookups through LMCache, hits and misses, against raw_block and fs_native, plus their effect on loads. The safety script and failure tests passed on the final code, on loop devices on a RAM disk. The speed hasn't been repeated on the EPYC node.
ci.yml never touched lmcache-connector/, so a green check said nothing about it. The steps are in the existing job, so the check that main requires covers them. The safety script still isn't in CI, because it needs LMCache and a running daemon.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Not ready to merge: see Before merging. Related to #4.
What this adds
An LMCache L2 connector in Rust, loaded through LMCache's
native_plugintype, that stores KV cache through the HyperCache daemon and stripes it across NVMe drives.lmcache-connector/: the connector. Below LMCache's own wrapper it runs without Python or the interpreter lock, and every load is checked with CRC32. Any doubt about a block is a miss.control-plane/src/protocol.rs,server.rs:commit_writeacceptslocal_onlyandchunk_checksums, andlocatereportssync_statuson hits. All optional on the wire, so older clients aren't affected.client-py/: O_DIRECT writes to raw block devices, off by default.Reviewing
Of about 835 changed files, 798 are raw benchmark data under
benchmarks/results/lmcache_l2_raw/, about 73,000 of the 78,000 added lines. Start withlmcache-connector/src/, the twocontrol-planefiles, thenbenchmarks/results/lmcache_rust_connector_nvme.md. Connector usage is inGETTING_STARTED.mdsection 9 andARCHITECTURE.mdsection 7.5.Results
Storage only, on bare metal with 4 Micron 7450 drives and no GPU, against LMCache
raw_blockon md RAID 0. 14 MiB objects, median of 3. Full results:benchmarks/results/lmcache_rust_connector_nvme.md.raw_blockon RAID 0raw_blockwithout RAID)raw_blockgot stuck near 15.2 GB/s in 4 of 6 RAID 0 runs on 4 drives. The cause isn't known and could be our array setup, so don't read this as a general fault ofraw_block.raw_block's 4.34 and 3.91. This hasn't been repeated on the bare-metal node.Testing
scripts/e2e/lmcache_l2_e2e.py): 30 of 30 on LMCache 0.5.5 and 0.5.6, on 1 and 2 devices, on the final code.kill -9during loads, stores and lookups, restarts, a frozen daemon, reused blocks, corruption, a capacity limit and a full device. A load was always a miss or the right bytes.The final-code safety and failure runs used loop devices on a RAM disk, with lower volumes to fit. Earlier rounds of the safety and failure tests ran on the real Micron drives, before the batched lookup change. The upgrade test also ran there, and the daemon hasn't changed since.
Before merging
benchmarks/results/lmcache_rust_connector_xeon.md.release.yml.ci.yml).Known limits
fs_nativeon multiple drives, or run the failure tests onraw_blockorfs_native, so these results don't show that HyperCache is safer than either.