Skip to content

[GUIDE] Interoperability with Polars, DuckDB, and PyArrow (Zero-Copy Analytics) #717

Description

@FrancescAlted

Blosc2 provides zero-copy Apache Arrow interoperability via CTable.to_arrow(), from_arrow(), and the PyCapsule protocol. Some data teams may want to leverage Blosc2 for high-ratio compressed columnar storage while running analytical queries using modern query engines like DuckDB and Polars.

Suggested Work

Create doc/guides/interop_arrow_duckdb_polars.md and link it from doc/guides/index.rst.

Provide runnable code examples showing:

  1. PyArrow Round-trip: Converting a blosc2.CTable to a pyarrow.Table / RecordBatch and back with zero copying where supported.
  2. Direct SQL queries with DuckDB: Querying Blosc2 tables using DuckDB's Arrow scanner:
    import duckdb
    import blosc2
    
    table = blosc2.open("dataset.b2z")
    res = duckdb.arrow(table.to_arrow()).filter("price > 100").aggregate("avg(score)").df()
  3. Polars Integration: Ingesting a Blosc2 table into Polars (pl.from_arrow(table.to_arrow())) and converting Polars DataFrames into Blosc2 CTables.
  4. Performance & Memory Comparisons: Illustrating memory usage during queries on compressed Blosc2 tables vs. raw in-memory DataFrames.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions