A major capability of Blosc2 is processing datasets that far exceed physical RAM using mmap_mode="r" or "r+" in blosc2.open(). While mentioned in doc/guides/optimization_tips.md, there is no dedicated guide explaining how memory-mapped arrays interact with the operating system page cache, two-level chunking, and multi-threaded decompression.
Suggested Work
Create doc/guides/memory_mapped_storage.md and link it from doc/guides/index.rst.
The guide should cover:
- The Out-of-Core Concept: How
mmap_mode allows opening a 100+ GB array on a machine with 16 GB RAM with near-zero initial memory overhead.
- Memory Profiling: Concrete code comparing standard reads vs.
mmap_mode="r" with peak memory measurements.
- Modes: Explaining read-only (
"r"), copy-on-write ("c"), and read-write ("r+") semantics.
- Chunk and Block Sizing for mmap: How block sizes affect decompression locality when paging from disk cache.
- Concurrency with mmap: Concurrently reading memory-mapped containers across multiple processes.
A major capability of Blosc2 is processing datasets that far exceed physical RAM using
mmap_mode="r"or"r+"inblosc2.open(). While mentioned in doc/guides/optimization_tips.md, there is no dedicated guide explaining how memory-mapped arrays interact with the operating system page cache, two-level chunking, and multi-threaded decompression.Suggested Work
Create
doc/guides/memory_mapped_storage.mdand link it fromdoc/guides/index.rst.The guide should cover:
mmap_modeallows opening a 100+ GB array on a machine with 16 GB RAM with near-zero initial memory overhead.mmap_mode="r"with peak memory measurements."r"), copy-on-write ("c"), and read-write ("r+") semantics.