Skip to content

[UTILS] Bi-directional Zarr conversion: Support importing Zarr to .b2nd #721

Description

@FrancescAlted

parquet-to-blosc2 supports bi-directional conversion (--export converts Blosc2 back to Parquet). In contrast, b2nd-to-zarr only converts from Blosc2 .b2nd to Zarr.

Users with large existing Zarr datasets (v2 or v3) currently cannot convert them into .b2nd format using the CLI.

Suggested Work

  1. Extend src/blosc2/cli/b2nd_to_zarr.py to support reverse conversion:
    • Add a --reverse / --export flag (or provide a secondary entrypoint script zarr-to-b2nd).
    • Use blosc2.ZarrNDSource or the zarr package to stream Zarr chunks into a target .b2nd NDArray in memory-bounded batches (--buffer-size).
    • Map Zarr chunk dimensions to Blosc2 chunks, and compute appropriate Blosc2 block sizes.
    • Preserve or translate compression parameters (codecs, shuffle filters) where possible.
  2. Add end-to-end roundtrip tests in tests/test_cli_b2nd_to_zarr.py.
  3. Document the reverse conversion command in doc/guides/b2nd_to_zarr.md.
  4. Analyze pros and cons of each format.

Bonus points: try to use a public Zarr dataset to offer a better overview in real situations.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions