Replace the remote HDF5 reference translation layer with native HDF5 indexing and direct fsspec range reads

doc/getting_started/installation.rst

@@ -44,9 +44,8 @@ grouped into *extras* that you opt into with the ``blosc2[extra]`` syntax:
      - Lazy Zarr sources and the ``b2nd-to-zarr`` (or ``blosc2-to-zarr``)
        converter (``zarr``).
    * - ``hdf5``
-     - Reading HDF5 datasets lazily as virtual arrays: ``h5py`` for local
-       files, ``kerchunk`` for remote ones (``kerchunk``, ``h5py``,
-       ``hdf5plugin``).
+     - Reading HDF5 datasets lazily as virtual arrays, locally or through
+       fsspec (``h5py``, ``hdf5plugin``).
    * - ``fsspec``
      - Reading and writing single-file containers through any `fsspec
        <https://filesystem-spec.readthedocs.io>`_ URL.  The HTTP(S) driver is

doc/guides/b2view.rst

@@ -97,7 +97,7 @@ objects and chunks. Small objects may fit entirely within a bounded opening read
 ``--profile`` and ``--endpoint-url`` are optional; when omitted, the S3 backend
 uses its normal credential and endpoint configuration. Install ``blosc2[tui]``
 and ``blosc2[fsspec]`` plus ``s3fs`` for S3 access. Zarr requires
-``blosc2[zarr]``; HDF5 requires ``blosc2[hdf5]`` (Kerchunk, h5py, and Zarr).
+``blosc2[zarr]``; HDF5 requires ``blosc2[hdf5]`` (h5py and optional filter plugins).
 B2Z browsing does not require Zarr or HDF5 dependencies.
 
 Format details and limits:
@@ -115,13 +115,12 @@ Format details and limits:
   support and LIST permission. Direct arrays do not require listing their parent.
   Empty groups and attributes are preserved. Unknown codecs and unsupported
   dtypes remain visible; preview support follows the existing Zarr array reader.
-* **HDF5:** Kerchunk translates metadata once per session and all selected leaves
-  reuse those references. Translation can enumerate many chunk references and
-  inline small values; it avoids full-file localization, but is not a constant-cost
-  operation. Empty groups and attributes are preserved. Failed dataset translations
-  become unavailable nodes without hiding supported siblings. The view covers
-  Kerchunk's representation: hard-link aliases may be omitted, and soft/external
-  links and group cycles are not followed.
+* **HDF5:** h5py builds a native metadata and chunk-range index once per session,
+  and all selected leaves reuse it. Indexing can enumerate many allocated chunks;
+  it avoids full-file localization, but is not a constant-cost operation. Empty
+  groups and attributes are preserved. Unsupported datasets remain unavailable
+  nodes without hiding supported siblings. Hard-link aliases may be omitted, and
+  soft/external links and group cycles are not followed.
 
 These internal browser adapters do not change the array-only contract of
 ``blosc2.open(..., lazy=True)`` or add a persisted RemoteArray hierarchy descriptor.

doc/guides/remote_arrays.md

@@ -36,7 +36,7 @@ The argument passed to {func}`blosc2.open` selects the route:
 | A URL string such as `s3://...` or `https://...`                              | fsspec   | A byte-addressable, standalone `.b2nd` file             |
 | A URL containing a `.b2z` path component, or `source_format="b2z"`            | B2Z      | One immutable external NDArray leaf in a `.b2z` archive |
 | A URL containing a `.zarr` path component                                     | Zarr     | One immutable Zarr v2 or v3 array                       |
-| A URL containing a `.h5` or `.hdf5` path component, or `source_format="hdf5"` | HDF5     | One immutable HDF5 dataset (h5py locally, kerchunk remotely) |
+| A URL containing a `.h5` or `.hdf5` path component, or `source_format="hdf5"` | HDF5     | One immutable HDF5 dataset (h5py locally, native range index remotely) |
 | A {ref}`URLPath`                                                              | Caterva2 | One array-like dataset on a Caterva2 server             |
 
 ```python
@@ -67,7 +67,7 @@ h1 = blosc2.open(
 store_b2z = blosc2.RemoteStore("s3://bucket/hierarchy.b2z")
 store_h5 = blosc2.RemoteStore("https://datasets.example.org/hierarchy.h5")
 
-# Reopen an exported reference snapshot (.b2z):
+# Reopen an exported portable snapshot (.b2z):
 store_snap = blosc2.open("snapshot.b2z")
 
 a1.shape, a1.dtype  # metadata is available immediately
@@ -89,13 +89,14 @@ Converted Blosc2 chunks are cached under an immutable source contract, so publis
 Remote HDF5 needs `pip install "blosc2[hdf5,fsspec]"`.
 HTTP/HTTPS works directly; cloud stores require their protocol driver (`s3fs` for S3, etc.).
 Datasets can be specified using standard slash syntax (`file.h5/d0/d1/a2`), the double-colon separator (`file.h5::d0/d1/a2`), or the `dataset="d0/d1/a2"` parameter.
-Pre-indexing is performed via `kerchunk`. When opening a single {ref}`RemoteArray`, the resulting reference map is cached inside the array carrier (`schunk.vlmeta["hdf5-refs"]`). When using {ref}`RemoteStore`, indexing is performed once for the entire container and shared across all leaves and sessions.
+Remote pre-indexing uses h5py to record dataset metadata and allocated chunk byte ranges. When opening a single {ref}`RemoteArray`, the native index is cached inside the array carrier (`schunk.vlmeta["hdf5-index"]`). When using {ref}`RemoteStore`, indexing is performed once for the entire container and shared across all leaves and sessions. Uncompressed, deflate, shuffle, and Blosc2 pipelines are decoded directly after fsspec range reads. Other pipelines use a retained h5py reader, including filters registered by `hdf5plugin`.
 Use `blosc2.available_datasets(url)` to inspect datasets in an HDF5 container.
 
-Local HDF5 files use h5py directly, without kerchunk pre-indexing or Zarr/fsspec
-dependencies. For example, `blosc2.open("hierarchy.h5::/d0/a2")` reads the selected
+Local HDF5 files use h5py directly, without pre-indexing or an fsspec
+dependency. For example, `blosc2.open("hierarchy.h5::/d0/a2")` reads the selected
 dataset through h5py and caches converted Blosc2 chunks in memory. Explicit
-`refs=` inputs retain the reference-based reader, including for local files.
+`hdf5_index=` accepts a native HDF5 index, including for local files. Legacy
+HDF5 reference maps are rejected; omit `hdf5_index=` to regenerate the native index.
 
 `RemoteArray` assumes remote sources are immutable by default, avoiding a metadata request before every read.
 For a replaceable `.b2nd` or Caterva2 source, pass `assume_immutable=False` to refresh its identity and invalidate stale cached chunks before each operation.
@@ -151,8 +152,9 @@ What differs between the transports is the types of remote objects each can open
 
 `lazy=True` changes *when* data is fetched; it does not expand the underlying storage formats supported by either route.
 
-> [!TIP]
-> **Browse remote hierarchies**: To explore groups, inspect attributes, or preview arrays in remote `.b2z`, `.zarr`, or `.h5` containers interactively in the terminal without downloading the complete container, use {doc}`b2view <b2view>` (e.g. `b2view s3://bucket/hierarchy.b2z`). To navigate containers programmatically in Python, use {ref}`RemoteStore`.
+```{tip}
+**Browse remote hierarchies**: To explore groups, inspect attributes, or preview arrays in remote `.b2z`, `.zarr`, or `.h5` containers interactively in the terminal without downloading the complete container, use {doc}`b2view <b2view>` (e.g. `b2view s3://bucket/hierarchy.b2z`). To navigate containers programmatically in Python, use {ref}`RemoteStore`.
+```
 
 ## Explore remote hierarchies with RemoteStore
 
@@ -203,7 +205,7 @@ with blosc2.RemoteStore(
 ```
 
 When reopening the same store later with the same `cache_dir`:
-- Discovery metadata (such as B2Z member offsets or HDF5 Kerchunk reference maps) is restored from local disk, avoiding repeated remote translation scans. `store.metadata_bytes` reports the encoded manifest size.
+- Discovery metadata (such as B2Z member offsets or native HDF5 indexes) is restored from local disk, avoiding repeated remote scans. `store.metadata_bytes` reports the encoded manifest size.
 - Retained leaf chunks are available immediately from disk without network transfers.
 - Single-owner locks ensure that concurrent processes do not corrupt the shared cache.
 
@@ -468,7 +470,7 @@ with blosc2.RemoteStore("https://datasets.example.org/data.h5") as store:
     store["experiment"].save("experiment_sub.b2z")
 ```
 
-- **Portable reference**: The `.b2z` archive contains the discovered hierarchy, attributes, and source locators (such as the HDF5 Kerchunk reference map or B2Z member offsets), but no secrets or credentials.
+- **Portable reference**: The `.b2z` archive contains the discovered hierarchy, attributes, and source locators (such as the native HDF5 index or B2Z member offsets), but no secrets or credentials.
 - **`include_cache=True` (default)**: Bundles warm cached chunks along with metadata so reading previously fetched slices requires zero network traffic.
 - **`include_cache=False`**: Omits cached payload chunks, producing a minimal reference archive for remote streaming.
 - **Subtree export**: Calling `save()` on a group view exports that subtree with relative child keys and the appropriate source root.
@@ -517,8 +519,9 @@ store.save("writable.b2z", mutable=True)
 | **Immutable** (`mutable=False`, default) | Reads directly in-place from `.b2z` without disk writes. Safe on read-only media (`chmod 0o444`). | Fetched transiently into RAM to satisfy the read; never written to disk or the archive. | `fetch()`, `afetch()`, `trim_cache()`, and `refresh()` are disallowed. |
 | **Mutable** (`mutable=True`) | Staged into an independent writable runtime cache directory. Original `.b2z` stays untouched. | Fetched and cached to disk under standard LRU eviction rules. | Fully supported. Can be opened with a smaller budget, trimming excess chunks. |
 
-> [!TIP]
-> Use **immutable snapshots** (`mutable=False`) for sharing reproducible, read-only reference archives or distributing datasets that should never modify local storage. Use **mutable snapshots** (`mutable=True`) when users should be able to expand the local cache with newly fetched regions over time.
+```{tip}
+Use **immutable snapshots** (`mutable=False`) for sharing reproducible, read-only reference archives or distributing datasets that should never modify local storage. Use **mutable snapshots** (`mutable=True`) when users should be able to expand the local cache with newly fetched regions over time.
+```
 
 ## Retrieve scattered points
 

doc/reference/hdf5ndsource.rst

@@ -4,16 +4,17 @@ HDF5NDSource
 ============
 
 ``HDF5NDSource`` exposes an HDF5 dataset through :ref:`ProxyNDSource`.
-Local files use ``h5py`` directly, without scanning the file with ``kerchunk``.
-Remote files and explicit ``refs=`` inputs use ``kerchunk`` reference maps.
+Local files use ``h5py`` directly. Remote files are scanned once with ``h5py``
+to build a native byte-range index, which can also be supplied through ``hdf5_index=``.
 Individual chunks are fetched on demand
 and converted to Blosc2-compressed chunks stored in the surrounding
 :ref:`Proxy` or :ref:`RemoteArray` cache.
 
 The source is assumed immutable (``assume_immutable=True``). It supports fixed-size
 boolean, integer, floating-point, complex, and fixed-length string arrays.
-HDF5 filters such as Blosc2 (via ``hdf5plugin``), gzip, and uncompressed datasets
-are supported.
+Uncompressed, gzip/deflate, shuffle, and Blosc2 chunks use direct range reads.
+Other filter pipelines fall back to retained ``h5py`` dataset reads;
+``hdf5plugin`` enables its additional registered filters.
 
 Local reads require ``h5py``; ``hdf5plugin`` enables additional HDF5 filters.
 Install the full HDF5 support with ``pip install "blosc2[hdf5]"``. Remote datasets also

doc/reference/remotearray.rst

@@ -58,10 +58,10 @@ the double-colon separator (``.../file.h5::dataset``), or the ``dataset="dataset
 Zarr containers similarly accept all three forms (``.../file.zarr/dataset``, ``.../file.zarr::dataset``,
 or ``dataset="dataset"``).
 HDF5 datasets on a local path are read directly with ``h5py``; remote HDF5 files use
-``kerchunk`` metadata pre-indexing. Like Zarr, HDF5 sources
+native metadata pre-indexing with ``h5py`` and byte ranges through fsspec. Like Zarr, HDF5 sources
 are assumed immutable (``assume_immutable=True``); mutable HDF5 sources are not supported.
-Pre-computed kerchunk references can be supplied via ``refs`` to avoid remote scanning
-(or to use the reference reader for a local file).
+Pre-computed native HDF5 indexes can be supplied via ``hdf5_index`` to avoid remote scanning
+(including when opening a local file through the indexed reader).
 
 .. code-block:: python
 
@@ -91,8 +91,8 @@ the same three addressing forms:
 
 Use ``source_format="b2z"`` for suffix-free archive URLs. The dataset is a logical
 tree key without the member's ``.b2nd`` suffix. The native Blosc2 reader preserves
-source chunks, blocks, dtype, and compression parameters; no kerchunk, Zarr, or
-HDF5 dependencies are needed. Install the fsspec extra and the protocol backend.
+source chunks, blocks, dtype, and compression parameters; no Zarr or HDF5
+dependencies are needed. Install the fsspec extra and the protocol backend.
 
 Opening reads the ZIP directory and selected member's headers. Directory cost
 scales with archive member count. An 8 KiB archive tail and 16 KiB member prefix

doc/reference/remotestore.rst

@@ -5,7 +5,7 @@ RemoteStore
 
 ``RemoteStore`` discovers a read-only B2Z, Zarr or HDF5 hierarchy and returns
 :ref:`RemoteArray` leaves. Groups and arrays share one source session: a B2Z
-archive, an HDF5 reference map, or a Zarr store. Zarr listing remains lazy.
+archive, a native HDF5 index, or a Zarr store. Zarr listing remains lazy.
 
 The default ``CachePolicy.MEMORY`` shares a 256 MiB allowance across all leaves.
 Set ``max_cache_bytes`` to a positive integer to change it. ``CachePolicy.NONE``
@@ -62,7 +62,7 @@ Closing a handle, or exiting its context, releases its ownership. Existing child
 handles remain usable until closed or garbage-collected. The last handle closes
 the owned archive/store wrappers and private HTTP/S3 transport sessions. Operations on an explicitly closed handle raise ``RuntimeError``.
 Standalone ``RemoteArray`` exports remain self-contained references, including
-the HDF5 reference map when applicable.
+the native HDF5 index when applicable.
 
 ``b2view`` uses ``RemoteStore`` for remote hierarchies with one 64 MiB MEMORY
 allowance, and ``RemoteArray`` for selected or directly opened leaves. Switching
@@ -85,7 +85,7 @@ the operating system releases the lock after a process exits or crashes.
 
 Reopening restores all previously created leaf caches and trims them against the
 new aggregate allowance before returning. The manifest preserves B2Z directory
-and bounded metadata reads, one HDF5 reference map, and lazily discovered Zarr
+and bounded metadata reads, one native HDF5 index, and lazily discovered Zarr
 metadata. Metadata reads can contain small inline values or incidental bytes in
 bounded prefixes; they are separate from evictable payload. ``metadata_bytes``
 is the encoded manifest size, and is zero without a disk manifest. Credentials