Skip to content

[DOC] Container Architecture & Chunking Sizing Guide (Decision Matrix) #722

Description

@FrancescAlted

New contributors and users face two key architectural questions when getting started:

  1. Which container should I choose? (NDArray vs CTable vs SChunk vs TreeStore vs DictStore vs BatchArray).
  2. How do I choose chunk and block shapes? (Balancing compression ratio, CPU cache locality, slicing granularity, and storage overhead).

While issue #651 addresses a terminology glossary, there is no practical decision matrix or chunking rules-of-thumb guide in the documentation.

Suggested Work

Create a Markdown guide at doc/getting_started/choosing_containers_and_chunking.md (and link it from doc/getting_started/index.rst).

The guide should feature:

  1. Container Decision Matrix: A structured table comparing:
    • Data Geometry (multidimensional, tabular/relational, key-value, hierarchical tree)
    • Read/write mutability and append performance
    • Query capabilities (expression evaluation, SQL/where filtering, coordinate slicing)
    • On-disk layout (.b2nd, .b2frame, .b2z, .b2d)
  2. Chunking vs. Blocking Rules of Thumb:
    • Clear explanation of the two-level chunking hierarchy: chunks represent disk/network transfer units (typically 1 MB – 64 MB), whereas blocks represent CPU cache units (typically 64 KB – 1 MB to fit L2/L3 caches).
    • Practical guidelines for 1D, 2D, and 3D data geometries.
    • Trade-offs: large chunks (better compression) vs. small chunks (faster random slicing).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions