Skip to content

perf(arrow): prune IN predicates with row-group stats - #705

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:in-minmax-pruning
Open

perf(arrow): prune IN predicates with row-group stats#705
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:in-minmax-pruning

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

IN min/max pruning is opt-in in the shared statistics evaluator. Parquet page statistics enabled it in #704, but Parquet and Mosaic row-group statistics still used the default fail-open behavior. As a result, non-matching row groups could not be skipped, including in Parquet files without page indexes.

Brief change log

  • Enable min/max pruning for IN predicates using Parquet and Mosaic row-group statistics.
  • Preserve fail-open behavior for missing, inverted, or incompatible statistics.
  • Add regression tests for row-group selection and avoided Mosaic column reads.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex

API and Format

Documentation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant