Add remote CTable access and unify remote caching and persistence

[FrancescAlted]

This PR adds read-only access to CTable datasets in remote B2Z archives and local or remote PyTables/HDF5 sources. Tables can be opened, queried, and exported on demand without first materializing the entire dataset locally.

Main changes:

• Introduce RemoteCTable, supporting fixed-width, string, binary, list, dictionary, and structured columns, including null masks, selections, views, and pandas/Arrow exports.
• Extend blosc2.open() to discover tables, arrays, and groups consistently, with path= selection and support for source-bound CTable columns.
• Use supported remote indexes through lazy sidecar reads, and import supported PyTables full indexes for indexed queries.
• Unify source metadata, attributes, traffic accounting, cache management, refresh, and lifetime handling through the common RemoteObject API.
• Add persistent, process-shared caches with aggregate size limits, bounded concurrent reads, and recovery after interrupted writers.
• Support portable remote references with optional retained caches, alongside explicit materialization into independent local objects. Preserve attributes, nested stores, and table indexes across these operations.
• Extend ListArray with nullable elements, nested lists, contains/overlaps predicates, and optional scalar membership indexes.
• Expand documentation, examples, benchmarks, and regression coverage, and reduce overhead in several slow tests.

Compatibility notes:

• RemoteCTable.save() now writes a remote reference; use materialize(), copy(), to_b2z(), or to_b2d() for independent local data.
• New list arrays default to batches of 2,048 rows. Explicit batch_rows=None preserves caller-controlled batching; existing persisted arrays remain readable.
• Sparse table/store caches default to an aggregate 256 MiB limit; max_cache_bytes=None allows unlimited caching.
• Replacing an existing reference destination requires explicit overwrite=True.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues affect remote decoding, cache invalidation, materialization, and API behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 4 · 1

Open (5)

• Shape product overflow enables oversized allocation · New
• Linked nested stores remain stale after generation reload · New
• Batch caches persist stale data after generation reload · New
• Materialization creates an unbounded in-memory table copy · New
• Missing type validation for dataset selector · New

What changed in this PR

This PR adds remote CTable access, unified remote caching and persistence, nested-store support, and enhanced ListArray functionality. The review identified unresolved issues involving remote decoding, cache invalidation, materialization, selector handling, and examples.

Changes:

• Adds remote/local CTable discovery, querying, indexes, exports, and source-bound columns.
• Introduces shared caches, portable references, nested stores, refresh, and materialization.
• Extends nullable/nested lists and membership indexes, with supporting tests and documentation.

File	Reviewed changes
todo/​change-dataset-path.md	Documents path= alias behavior.
tests/​test_zarr_source.py	Tests local Zarr disk caching.
tests/​test_tree_store.py	Tests remote references and validation.
tests/​test_python_blosc.py	Refines leak detection coverage.
tests/​test_locking.py	Tests locking and concurrency behavior.
tests/​test_list_array.py	Tests nullable, nested, and membership lists.
tests/​test_fsspec.py	Tests shared and source caching.
tests/​test_dataset_path.py	Tests selector aliases and dispatch.
tests/​ndarray/​test_proxy.py	Expands chunk-boundary coverage.
tests/​ndarray/​test_indexing.py	Tests membership-index validation.
tests/​ndarray/​test_c2array_blocks.py	Tests shared Caterva2 caching.
tests/​ctable/​test_varlen_schema_compiler.py	Tests nested and legacy schemas.
tests/​ctable/​test_table_persistency.py	Tests table persistence and materialization.
tests/​ctable/​test_remote_pytables_interop.py	Tests PyTables interoperability.
tests/​ctable/​test_remote_columns.py	Tests source-bound remote columns.
tests/​ctable/​test_column.py	Tests index reporting.
tests/​conftest.py	Adds bounded remote test fixtures.
tests/​b2view/​test_hierarchy.py	Tests remote table and group browsing.
src/​blosc2/​tree_store.py	Adds remote references and materialization; cache-policy persistence requires correction.
src/​blosc2/​store_materialize.py	Implements recursive materialization; large tables should be processed in bounded batches.
src/​blosc2/​schema.py	Adds nested lists and batching.
src/​blosc2/​schema_compiler.py	Supports nested and legacy schema validation.
src/​blosc2/​scalar_array.py	Adds lifecycle and write checks.
src/​blosc2/​remote_store_cache.py	Implements shared-cache lifecycle; generation reload must invalidate stale caches and nested owners.
src/​blosc2/​remote_source_cache.py	Adds persistent source caching.
src/​blosc2/​remote_object.py	Defines the common remote-object API.
src/​blosc2/​remote_batch.py	Adds remote batch access and caching.
src/​blosc2/​proxy.py	Documents remote proxy usage.
src/​blosc2/​proxy_source.py	Adds lazy frame-index reads and size metadata.
src/​blosc2/​ndarray.py	Documents membership-index limitations.
src/​blosc2/​msgpack_utils.py	Adds remote decoding; shape sizes must be bounded before allocation.
src/​blosc2/​dictionary_column.py	Adds read-only dictionary-column handling.
src/​blosc2/​dict_store.py	Handles archived remote array members.
src/​blosc2/​ctable_indexing.py	Implements membership indexes and remote sidecars.
src/​blosc2/​core.py	Resolves dataset/path selectors and cache paths.
src/​blosc2/​blosc2_ext.pyx	Releases the GIL during locked opens.
src/​blosc2/​b2view/​model.py	Supports remote-store nodes.
src/​blosc2/​b2view/​app.py	Updates remote-store navigation.
src/​blosc2/​_utf8_array.py	Adds remote UTF-8 lifecycle handling.
src/​blosc2/​__init__.py	Exports new remote APIs and index kinds.
RELEASE_NOTES.md	Documents compatibility and feature changes.
pyproject.toml	Adds the PyTables test dependency.
plans/​remote-nested-store.md	Documents nested-store design.
plans/​remote-ctable-v2.md	Documents remote CTable design.
plans/​remote-ctable-save.md	Documents reference-saving behavior.
plans/​remote-ctable-indexes.md	Documents remote index support.
plans/​local-cache-dir.md	Documents local cache behavior.
examples/​vlmeta.py	Updates metadata examples.
examples/​tree-store.py	Updates TreeStore metadata examples.
examples/​tree-store-blog.py	Updates metadata documentation examples.
examples/​remote/​s3-access.py	Updates remote metadata display.
examples/​remote/​nested-store.py	Demonstrates nested references and materialization; reopening needs correction.
examples/​ref-object.py	Updates reference metadata examples.
examples/​ndarray/​swmr-enlarge.py	Updates SWMR metadata usage.
examples/​ndarray/​swmr-enlarge-bars.py	Updates SWMR metadata usage.
examples/​ndarray/​proxy-ndarray.py	Updates metadata inspection.
examples/​embedded-expr-udf-b2z.py	Updates expression metadata access.
examples/​embed-store.py	Updates embedded-store metadata usage.
examples/​dict-store.py	Updates dictionary-store metadata usage.
examples/​ctable/​remote_pytables.py	Adds a PyTables remote-access example.
examples/​ctable/​remote_columns.py	Adds a source-bound column example.
doc/​reference/​remotestore.rst	Documents remote stores and nested references.
doc/​reference/​remoteobject.rst	Documents the common remote API.
doc/​reference/​remotectable.rst	Documents RemoteCTable.
doc/​reference/​remotearray.rst	Documents remote-array caching and indexes.
doc/​reference/​list_array.rst	Documents list batching and predicates.
doc/​reference/​hdf5ndsource.rst	Documents HDF5 index helpers.
doc/​reference/​ctable.rst	Documents source-bound and list columns.
doc/​reference/​classes.rst	Adds remote classes to API listings.
doc/​guides/​remote_tables.md	Adds remote table usage guidance.
doc/​guides/​index.rst	Registers new guides.
doc/​development/​remote_cache_design.md	Documents cache architecture.
doc/​development/​index.rst	Registers cache design documentation.
bench/​remote_hdf5_source_cache.py	Adds source-cache benchmarking.
bench/​remote_ctable_metadata.py	Adds metadata concurrency benchmarking.
bench/​remote_ctable_metadata.md	Documents benchmark methodology; the sample command needs correction.
bench/​remote_ctable_indexes.py	Adds indexed-query benchmarking.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Four moderate findings remain: the nested-store example, remote sparse-cache options, membership-index materialization, and temporary-array cleanup.

Review effort: Lite
Findings: None

Resolved since last review (5)

• Materialization creates an unbounded in-memory table copy
• Batch caches persist stale data after generation reload
• Linked nested stores remain stale after generation reload
• Shape product overflow enables oversized allocation
• Missing type validation for dataset selector

Previously missed (1)

In code that hasn't changed since last review

Forward storage_options in sparse-cache factory

src/​blosc2/​remote_ctable.py:176

Unlike RemoteStore.with_sparse_cache(), this public sparse-cache factory does not accept storage_options, so callers opening S3/HTTP tables with a custom endpoint, profile, or credentials cannot use it (TypeError), even though the wrapped constructor supports those options. Add the keyword and forward it to RemoteStore.with_sparse_cache().