Blosc2 provides zero-copy Apache Arrow interoperability via CTable.to_arrow(), from_arrow(), and the PyCapsule protocol. Some data teams may want to leverage Blosc2 for high-ratio compressed columnar storage while running analytical queries using modern query engines like DuckDB and Polars.
Suggested Work
Create doc/guides/interop_arrow_duckdb_polars.md and link it from doc/guides/index.rst.
Provide runnable code examples showing:
- PyArrow Round-trip: Converting a
blosc2.CTable to a pyarrow.Table / RecordBatch and back with zero copying where supported.
- Direct SQL queries with DuckDB: Querying Blosc2 tables using DuckDB's Arrow scanner:
import duckdb
import blosc2
table = blosc2.open("dataset.b2z")
res = duckdb.arrow(table.to_arrow()).filter("price > 100").aggregate("avg(score)").df()
- Polars Integration: Ingesting a Blosc2 table into Polars (
pl.from_arrow(table.to_arrow())) and converting Polars DataFrames into Blosc2 CTables.
- Performance & Memory Comparisons: Illustrating memory usage during queries on compressed Blosc2 tables vs. raw in-memory DataFrames.
Blosc2 provides zero-copy Apache Arrow interoperability via
CTable.to_arrow(),from_arrow(), and the PyCapsule protocol. Some data teams may want to leverage Blosc2 for high-ratio compressed columnar storage while running analytical queries using modern query engines like DuckDB and Polars.Suggested Work
Create
doc/guides/interop_arrow_duckdb_polars.mdand link it fromdoc/guides/index.rst.Provide runnable code examples showing:
blosc2.CTableto apyarrow.Table/RecordBatchand back with zero copying where supported.pl.from_arrow(table.to_arrow())) and converting Polars DataFrames into Blosc2 CTables.