Skip to content

refactor: build Iceberg test metadata at runtime - #700

Merged
alexanderbianchi merged 8 commits into
datafusion-contrib:iceberg-0.10from
alexanderbianchi:codex/iceberg-runtime-metadata
Sep 7, 2026
Merged

refactor: build Iceberg test metadata at runtime#700
alexanderbianchi merged 8 commits into
datafusion-contrib:iceberg-0.10from
alexanderbianchi:codex/iceberg-runtime-metadata

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Tests can supply complete Iceberg table metadata, explicit table options, and in-memory fixture files through IcebergTestHarness::builder().

  • with_table_metadata accepts native Iceberg metadata without reconstructing its IDs, schemas, or snapshot history.
  • with_table_option replaces duplicated registration SQL in the selected-snapshot and invalid-snapshot-ID tests.
  • with_file(uri, bytes) overrides a fixture file, including manifests and manifest lists. Explicit files take precedence over generated metadata; unmodified paths fall back to the checked-in taxi fixture. An empty-table registration test verifies raw metadata override behavior.

build() now constructs the session and registers the table directly; there is no intermediate create() helper. The harness remains usable without distributed integration features.

Shared fixture helpers expose only the original taxi metadata and a native metadata builder starting before the first snapshot. Snapshot recipes stay test-local. The duplicated missing-summary JSON fixture is removed.

Rebased onto iceberg-0.10 at 6507bf8, including merged #687. The table-options cleanup and file-override support are separate commits. Test-case consolidation is intentionally kept in an independent PR.

Validation on #700 at bb3872e:

  • cargo test -p datafusion-distributed-iceberg --locked — 90 tests passed, including the doctest
  • cargo clippy -p datafusion-distributed-iceberg --all-targets --locked -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Comment thread iceberg/src/test_utils/harness.rs
@alexanderbianchi
alexanderbianchi force-pushed the codex/iceberg-runtime-metadata branch from 0f95ee8 to d4ea4f5 Compare September 5, 2026 15:41
}
}

enum FixtureInput<'a> {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First iteration required a duplicated check in each function above, i'd rather keep them 1 line and isolate the input to one place. If you think a struct for this is overkill happy to split it up into the impl Storage

@alexanderbianchi
alexanderbianchi marked this pull request as ready for review September 5, 2026 15:58
.remove("total-files-size")
.is_some()
);
let snapshot = Snapshot::builder()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we could do like .into_builder() on the current snapshot this wouldn't be a problem but the LoC for a new snapshot are needed right now.

Comment thread iceberg/src/test_utils/harness.rs
Comment thread iceberg/tests/external_table.rs Outdated
.await?;
let (_, batches) = harness.query("SELECT COUNT(*) AS trips FROM taxi").await?;

insta::assert_snapshot!(batches, @"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider something like

   async fn run(
       ctx: &SessionContext,
       query: &str,
   ) -> Result<(Arc<dyn ExecutionPlan>, Vec<RecordBatch>)> {
       let df = ctx.sql(query).await?;
       let plan = df.create_physical_plan().await?;
       let batches = collect(Arc::clone(&plan), ctx.task_ctx()).await?;
       Ok((plan, batches))
   }

from tests/multi_task_collect_join_repros.rs.

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 no blockers, mainly just questions

Comment thread iceberg/src/test_utils/fixture.rs Outdated
Comment thread iceberg/tests/external_table.rs Outdated
Comment thread iceberg/tests/statistics.rs
alexanderbianchi added a commit that referenced this pull request Sep 7, 2026
Make the context private and configure target partitions, scan byte budgets, and column statistics before building the session. Move scan and estimation plumbing into the harness so tests share its configuration.

Adapt task-count fixtures to #700's empty_taxi_metadata_builder rename. Keep distributed execution opt-in and preserve existing snapshots.
@alexanderbianchi
alexanderbianchi merged commit 42cb491 into datafusion-contrib:iceberg-0.10 Sep 7, 2026
32 checks passed
alexanderbianchi added a commit that referenced this pull request Sep 8, 2026
## Summary

Implements the Iceberg desired-task-count handler using the selected
snapshot's `total-files-size` summary, read from the coordinator-local
work-unit feed.

- Respect an explicitly selected snapshot, otherwise use the current
snapshot.
- Estimate tasks using the configured bytes per partition and target
partitions, with ceiling division and validation of zero divisors.
- Treat a table with no current snapshot as zero scan work; decline to
estimate when size metadata is missing, invalid, or unavailable on a
remote feed.
- Add focused tests using the runtime metadata fixtures from #700,
including historical snapshots, empty tables, malformed summaries, and
decoded remote feeds.

This is a whole-snapshot scan-work estimate, not a post-pruning byte
estimate.

Closes #605.

## Dependency / scope

Stacked directly on #700's `codex/iceberg-runtime-metadata`.

Named `test-case` cases cover current and selected snapshots, empty
tables, and missing or invalid file sizes. A separate
distributed-execution test follows `tests/task_estimator_test.rs`: it
executes a taxi aggregation through the in-memory workers. The core
harness and its existing tests remain available without the
`integration` feature. Distributed planning and worker setup are
explicit builder choices through `.with_workers(...)`, requiring
`integration`; CI enables that feature for the distributed test. The
construction cleanup belongs to #700, so this PR no longer introduces
and then removes `create()`. Existing plan and result snapshots remain
unchanged.

## Validation

- `cargo test -p datafusion-distributed-iceberg --locked` — 101 tests
passed, including harness-backed tests and the doctest.
- `cargo test -p datafusion-distributed-iceberg --features integration
--locked` — 102 tests passed, including distributed execution and the
doctest.
- `cargo clippy -p datafusion-distributed-iceberg --all-targets -- -D
warnings` — passed, also with `--all-features`.
- `cargo check -p datafusion-distributed-iceberg --lib` — passed without
the integration feature.
- `cargo fmt --all -- --check` — passed.
- `git diff --check origin/iceberg-0.10 HEAD` — passed.
alexanderbianchi added a commit that referenced this pull request Sep 8, 2026
Extends `roundtrips_data_source_plan`, the existing codec test from
#684, with two explicit storage properties supplied through #700's
harness builder. Checks their decoded values, including a
quote-containing value that exercises SQL literal escaping, while
retaining the existing schema, partitioning, fetch, feed, property-map,
and statistics assertions.

The diff against #700 remains 15 changed lines in
`iceberg/src/codec.rs`, with no new harness methods or test functions.

Stacked on #700, now rebased onto `iceberg-0.10` at `6507bf8` after #687
merged. Only #700 is a prerequisite.

Validation:
- `cargo test -p datafusion-distributed-iceberg --locked` — 90 tests
passed, including the codec roundtrip test and doctest
- `cargo clippy -p datafusion-distributed-iceberg --tests --locked -- -D
warnings`
- `cargo fmt --all -- --check`
- `git diff --check`
alexanderbianchi added a commit that referenced this pull request Sep 8, 2026
Reapplies the reviewed and merged changes from #703 to the intended
base, `iceberg-0.10`.

#703 was still targeting `codex/iceberg-runtime-metadata` when it
merged, after #700 had already merged into `iceberg-0.10`. Consequently,
its codec-test improvements landed only on the old topic branch.

This PR cherry-picks #703's merge commit (`eafc234`) onto
`iceberg-0.10`, without additional code changes. It retains the original
15-line diff in `iceberg/src/codec.rs`: explicit storage properties,
including a quote-containing value, and assertions that those properties
survive the codec round trip.

Validation:
- `cargo test -p datafusion-distributed-iceberg --locked --lib
roundtrips_data_source_plan` — passed.
- `cargo fmt --all -- --check` and `git diff --check` passed.

Targets `iceberg-0.10` directly; it does not depend on the separate
reapplication of #704.
alexanderbianchi added a commit that referenced this pull request Sep 8, 2026
Reapplies the reviewed and merged changes from #704 to the intended
base, `iceberg-0.10`.

#704 was still targeting `codex/iceberg-runtime-metadata` when it
merged, after #700 had already merged into `iceberg-0.10`. Consequently,
the estimator and harness changes landed only on the old topic branch.

This PR cherry-picks #704's merge commit (`d3ba554`) onto
`iceberg-0.10`. There are no additional code changes; the resulting tree
is identical to #704's final head (`12e1512`).

Includes the source-statistics-based task estimator, explicit estimation
tests, distributed execution coverage, and the private-context harness
with `configure_session(...)`.

Validation:
- `cargo test -p datafusion-distributed-iceberg --features integration
--locked --test desired_task_count` — all 9 tests passed.
- `cargo fmt --all -- --check` and `git diff --check` passed.
- Verified exact tree equality with #704's final head.

Targets `iceberg-0.10` directly; it does not depend on the separate
reapplication of #703.
alexanderbianchi added a commit that referenced this pull request Sep 9, 2026
## Less test code, stronger statistics coverage

Based directly on `iceberg-0.10` at `f9340ef`, including merged #715 and
#716. “Before” refers to that base. Only `iceberg/tests/statistics.rs`
changes, with a net reduction of 24 lines.

| Test / edge case | Before | After |
|---|---|---|
| Row count and full-scan byte size | Separate enabled/disabled test
bodies | Ordinary named tests share assertions and retain both modes |
| Full-schema column statistics | Vector length only, in another pair of
tests | Complete expected vector: populated metrics and unknown columns
|
| Reordered projection / nonconsecutive selected field IDs | Projected
vector length and row count | Exact association of metrics with
reordered columns using IDs 4, 1, and 5 |
| Null-count aggregation across files | **None** | Exact sum of 5 |
| Column-size aggregation across files | **None** | Inexact sums of 400
and 600 |
| Min/max bounds and scalar types | **None** | Int32 and Int64 bounds
checked with their precision; opposing extrema order exercises both
updating and retaining bounds |
| One file missing a column's null count | **None** | Aggregate stays
`Absent`, rather than treating missing as zero |
| Entirely missing column metrics | Only an all-empty-metrics fixture |
Unknown column alongside columns with known metrics |
| Column statistics disabled | Fixture had no metrics even when enabled
| Populated fixture must return unknown column statistics when disabled
|
| Explicit current-snapshot selection | Separate duplicate row-total
test | Fixture selects the snapshot explicitly in every matrix case |
| Missing snapshot summary totals | Present | Retained |
| Filter and projection/sort propagation | Present | Retained |
| `COUNT(*)` skips the scan | Duplicated enabled/disabled bodies and
snapshots | Both named cases retained, sharing assertions |
| Explain formatting | Two identical snapshots plus weak label checks |
One diagnostic snapshot; computed values verified by the matrix |

## Structure

- Four ordinary named Tokio tests cover full scan / reordered projection
× column statistics enabled / disabled. Two more named tests cover
`COUNT(*)` scan elimination. No `test_case` macros or dependency.
- Session setup uses the merged harness's `configure_session(...)`;
tests never access its private context.
- Native Iceberg writers create a manifest with two synthetic data-file
entries and a matching manifest list. #700's `with_file` supplies the
bytes to the harness.
- Assertions observe query-output statistics, removing the recursive
search/downcast helper and also checking projection propagation.
- `insta::allow_duplicates!` is needed only for the shared, identical
`COUNT(*)` inline snapshots.
- The fixture is planning-only: synthetic Parquet paths are not opened.
Multi-manifest merging, delete files, NDV, and verification against
actual Parquet contents are not claimed here.
- The genuinely historical-snapshot regression remains in #702; this PR
consolidates the current-snapshot-only check from #687.

## Validation

- `cargo test -p datafusion-distributed-iceberg --test statistics
--locked` — all 10 cases passed
- `cargo test -p datafusion-distributed-iceberg --locked` — 95 tests
passed, including the doctest
- `cargo test -p datafusion-distributed-iceberg --features integration
--locked` — 96 tests passed, including the doctest
- `cargo clippy -p datafusion-distributed-iceberg --all-targets --locked
-- -D warnings` — passed, also with `--all-features`
- `cargo fmt --all -- --check`
- `git diff --check`
- Mutation check: replacing computed column statistics with unknown
values fails both enabled cases; the other eight tests pass. Mutation
reverted.
- Bounds mutation check: keeping the first minimum and taking the last
maximum fails both enabled cases; the other eight tests pass. Mutation
reverted.
alexanderbianchi added a commit that referenced this pull request Sep 9, 2026
Rebased directly onto `iceberg-0.10` after #700, #705, #715, and #716
merged.

Strengthens the existing selected-snapshot statistics test rather than
adding overlapping coverage. A test-local helper uses native Iceberg
builders to add snapshot 42 while leaving the original taxi snapshot
current. The completed metadata and `iceberg.snapshot_id = 42` are
supplied through the shared harness.

The test reports `Exact(42)` rows and `Exact(4242)` bytes,
distinguishing the selected snapshot from the current snapshot's 175,000
rows and 4,480,382 bytes.

The summary totals are deliberately synthetic. The manifest still
describes the original taxi data, so this fixture tests statistics
planning rather than executing a 42-row scan. This limitation is
documented beside its construction.

Diff against `iceberg-0.10`: one test file, 45 added lines, no
production changes.

Validation:
- `cargo test -p datafusion-distributed-iceberg --locked --test
statistics reports_statistics_for_the_selected_snapshot` — passed
- `cargo fmt --all -- --check`
- `git diff --check`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants