Skip to content

[GUIDE] Zero-Copy Out-of-Core Processing with Memory-Mapped Files (mmap) #719

Description

@FrancescAlted

A major capability of Blosc2 is processing datasets that far exceed physical RAM using mmap_mode="r" or "r+" in blosc2.open(). While mentioned in doc/guides/optimization_tips.md, there is no dedicated guide explaining how memory-mapped arrays interact with the operating system page cache, two-level chunking, and multi-threaded decompression.

Suggested Work

Create doc/guides/memory_mapped_storage.md and link it from doc/guides/index.rst.

The guide should cover:

  1. The Out-of-Core Concept: How mmap_mode allows opening a 100+ GB array on a machine with 16 GB RAM with near-zero initial memory overhead.
  2. Memory Profiling: Concrete code comparing standard reads vs. mmap_mode="r" with peak memory measurements.
  3. Modes: Explaining read-only ("r"), copy-on-write ("c"), and read-write ("r+") semantics.
  4. Chunk and Block Sizing for mmap: How block sizes affect decompression locality when paging from disk cache.
  5. Concurrency with mmap: Concurrently reading memory-mapped containers across multiple processes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions