New contributors and users face two key architectural questions when getting started:
- Which container should I choose? (
NDArray vs CTable vs SChunk vs TreeStore vs DictStore vs BatchArray).
- How do I choose chunk and block shapes? (Balancing compression ratio, CPU cache locality, slicing granularity, and storage overhead).
While issue #651 addresses a terminology glossary, there is no practical decision matrix or chunking rules-of-thumb guide in the documentation.
Suggested Work
Create a Markdown guide at doc/getting_started/choosing_containers_and_chunking.md (and link it from doc/getting_started/index.rst).
The guide should feature:
- Container Decision Matrix: A structured table comparing:
- Data Geometry (multidimensional, tabular/relational, key-value, hierarchical tree)
- Read/write mutability and append performance
- Query capabilities (expression evaluation, SQL/where filtering, coordinate slicing)
- On-disk layout (
.b2nd, .b2frame, .b2z, .b2d)
- Chunking vs. Blocking Rules of Thumb:
- Clear explanation of the two-level chunking hierarchy: chunks represent disk/network transfer units (typically 1 MB – 64 MB), whereas blocks represent CPU cache units (typically 64 KB – 1 MB to fit L2/L3 caches).
- Practical guidelines for 1D, 2D, and 3D data geometries.
- Trade-offs: large chunks (better compression) vs. small chunks (faster random slicing).
New contributors and users face two key architectural questions when getting started:
NDArrayvsCTablevsSChunkvsTreeStorevsDictStorevsBatchArray).While issue #651 addresses a terminology glossary, there is no practical decision matrix or chunking rules-of-thumb guide in the documentation.
Suggested Work
Create a Markdown guide at
doc/getting_started/choosing_containers_and_chunking.md(and link it fromdoc/getting_started/index.rst).The guide should feature:
.b2nd,.b2frame,.b2z,.b2d)