Skip to content

refactor: build manifest statistics fixtures in memory - #706

Closed
alexanderbianchi wants to merge 1 commit into
datafusion-contrib:codex/iceberg-runtime-metadatafrom
alexanderbianchi:iceberg/test-manifest-statistics
Closed

refactor: build manifest statistics fixtures in memory#706
alexanderbianchi wants to merge 1 commit into
datafusion-contrib:codex/iceberg-runtime-metadatafrom
alexanderbianchi:iceberg/test-manifest-statistics

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #705, which combines the test consolidation and populated-manifest coverage in one PR based on #700. The combined version replaces weak checks instead of adding a parallel test suite, includes missing-metric cases, and has a net LoC reduction. See #705 for the before/after coverage grid.

Original proposal

Draft example of #700's generic with_file(uri, bytes) support. Adds two integration tests using manifests and a manifest list serialized by Iceberg's native writers into memory, without new harness methods or dependencies.

The fixture contains two data files with known metrics for vendor_id (Iceberg field ID 1) and passenger_count (ID 4). Tests assert:

  • null-count and byte-size aggregation across files;
  • merged min/max bounds and their precision;
  • preservation of metrics through reordered projection, using non-contiguous field IDs.

Unlike the checked-in taxi manifest, whose column metric maps are empty, this fixture produces populated column statistics. The tests use synthetic data-file paths and only plan queries; they do not claim to validate statistics against actual Parquet contents. File row counts and sizes sum to the existing snapshot summary totals.

Stacked on #700 (codex/iceberg-runtime-metadata). Missing-metric combinations, multiple manifests, delete files, and NDV metadata were outside this initial example.

Original validation

  • Both new tests pass with column statistics enabled.
  • Temporarily disabling column statistics makes both fail with Absent values; the mutation was reverted.
  • cargo test -p datafusion-distributed-iceberg --locked — 92 tests passed, including the doctest
  • cargo clippy -p datafusion-distributed-iceberg --tests --locked -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant