refactor: build Iceberg test metadata at runtime - #700
Merged
alexanderbianchi merged 8 commits intoSep 7, 2026
Merged
Conversation
alexanderbianchi
force-pushed
the
codex/iceberg-runtime-metadata
branch
from
September 5, 2026 15:32
d33982b to
0f95ee8
Compare
alexanderbianchi
commented
Sep 5, 2026
alexanderbianchi
force-pushed
the
codex/iceberg-runtime-metadata
branch
from
September 5, 2026 15:41
0f95ee8 to
d4ea4f5
Compare
alexanderbianchi
commented
Sep 5, 2026
| } | ||
| } | ||
|
|
||
| enum FixtureInput<'a> { |
Collaborator
Author
There was a problem hiding this comment.
First iteration required a duplicated check in each function above, i'd rather keep them 1 line and isolate the input to one place. If you think a struct for this is overkill happy to split it up into the impl Storage
alexanderbianchi
marked this pull request as ready for review
September 5, 2026 15:58
alexanderbianchi
commented
Sep 5, 2026
| .remove("total-files-size") | ||
| .is_some() | ||
| ); | ||
| let snapshot = Snapshot::builder() |
Collaborator
Author
There was a problem hiding this comment.
If we could do like .into_builder() on the current snapshot this wouldn't be a problem but the LoC for a new snapshot are needed right now.
alexanderbianchi
commented
Sep 5, 2026
This was referenced Sep 5, 2026
Closed
alexanderbianchi
force-pushed
the
codex/iceberg-runtime-metadata
branch
from
September 5, 2026 21:22
71a906b to
3445b6a
Compare
alexanderbianchi
force-pushed
the
codex/iceberg-runtime-metadata
branch
from
September 6, 2026 00:18
158065b to
cdf47f7
Compare
This was referenced Sep 6, 2026
alexanderbianchi
commented
Sep 6, 2026
| .await?; | ||
| let (_, batches) = harness.query("SELECT COUNT(*) AS trips FROM taxi").await?; | ||
|
|
||
| insta::assert_snapshot!(batches, @" |
Collaborator
Author
There was a problem hiding this comment.
Should we consider something like
async fn run(
ctx: &SessionContext,
query: &str,
) -> Result<(Arc<dyn ExecutionPlan>, Vec<RecordBatch>)> {
let df = ctx.sql(query).await?;
let plan = df.create_physical_plan().await?;
let batches = collect(Arc::clone(&plan), ctx.task_ctx()).await?;
Ok((plan, batches))
}from tests/multi_task_collect_join_repros.rs.
gabotechs
approved these changes
Sep 7, 2026
gabotechs
left a comment
Collaborator
There was a problem hiding this comment.
👍 no blockers, mainly just questions
alexanderbianchi
added a commit
that referenced
this pull request
Sep 7, 2026
Make the context private and configure target partitions, scan byte budgets, and column statistics before building the session. Move scan and estimation plumbing into the harness so tests share its configuration. Adapt task-count fixtures to #700's empty_taxi_metadata_builder rename. Keep distributed execution opt-in and preserve existing snapshots.
alexanderbianchi
merged commit Sep 7, 2026
42cb491
into
datafusion-contrib:iceberg-0.10
32 checks passed
alexanderbianchi
added a commit
that referenced
this pull request
Sep 8, 2026
## Summary Implements the Iceberg desired-task-count handler using the selected snapshot's `total-files-size` summary, read from the coordinator-local work-unit feed. - Respect an explicitly selected snapshot, otherwise use the current snapshot. - Estimate tasks using the configured bytes per partition and target partitions, with ceiling division and validation of zero divisors. - Treat a table with no current snapshot as zero scan work; decline to estimate when size metadata is missing, invalid, or unavailable on a remote feed. - Add focused tests using the runtime metadata fixtures from #700, including historical snapshots, empty tables, malformed summaries, and decoded remote feeds. This is a whole-snapshot scan-work estimate, not a post-pruning byte estimate. Closes #605. ## Dependency / scope Stacked directly on #700's `codex/iceberg-runtime-metadata`. Named `test-case` cases cover current and selected snapshots, empty tables, and missing or invalid file sizes. A separate distributed-execution test follows `tests/task_estimator_test.rs`: it executes a taxi aggregation through the in-memory workers. The core harness and its existing tests remain available without the `integration` feature. Distributed planning and worker setup are explicit builder choices through `.with_workers(...)`, requiring `integration`; CI enables that feature for the distributed test. The construction cleanup belongs to #700, so this PR no longer introduces and then removes `create()`. Existing plan and result snapshots remain unchanged. ## Validation - `cargo test -p datafusion-distributed-iceberg --locked` — 101 tests passed, including harness-backed tests and the doctest. - `cargo test -p datafusion-distributed-iceberg --features integration --locked` — 102 tests passed, including distributed execution and the doctest. - `cargo clippy -p datafusion-distributed-iceberg --all-targets -- -D warnings` — passed, also with `--all-features`. - `cargo check -p datafusion-distributed-iceberg --lib` — passed without the integration feature. - `cargo fmt --all -- --check` — passed. - `git diff --check origin/iceberg-0.10 HEAD` — passed.
alexanderbianchi
added a commit
that referenced
this pull request
Sep 8, 2026
Extends `roundtrips_data_source_plan`, the existing codec test from #684, with two explicit storage properties supplied through #700's harness builder. Checks their decoded values, including a quote-containing value that exercises SQL literal escaping, while retaining the existing schema, partitioning, fetch, feed, property-map, and statistics assertions. The diff against #700 remains 15 changed lines in `iceberg/src/codec.rs`, with no new harness methods or test functions. Stacked on #700, now rebased onto `iceberg-0.10` at `6507bf8` after #687 merged. Only #700 is a prerequisite. Validation: - `cargo test -p datafusion-distributed-iceberg --locked` — 90 tests passed, including the codec roundtrip test and doctest - `cargo clippy -p datafusion-distributed-iceberg --tests --locked -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check`
This was referenced Sep 8, 2026
alexanderbianchi
added a commit
that referenced
this pull request
Sep 8, 2026
Reapplies the reviewed and merged changes from #703 to the intended base, `iceberg-0.10`. #703 was still targeting `codex/iceberg-runtime-metadata` when it merged, after #700 had already merged into `iceberg-0.10`. Consequently, its codec-test improvements landed only on the old topic branch. This PR cherry-picks #703's merge commit (`eafc234`) onto `iceberg-0.10`, without additional code changes. It retains the original 15-line diff in `iceberg/src/codec.rs`: explicit storage properties, including a quote-containing value, and assertions that those properties survive the codec round trip. Validation: - `cargo test -p datafusion-distributed-iceberg --locked --lib roundtrips_data_source_plan` — passed. - `cargo fmt --all -- --check` and `git diff --check` passed. Targets `iceberg-0.10` directly; it does not depend on the separate reapplication of #704.
alexanderbianchi
added a commit
that referenced
this pull request
Sep 8, 2026
Reapplies the reviewed and merged changes from #704 to the intended base, `iceberg-0.10`. #704 was still targeting `codex/iceberg-runtime-metadata` when it merged, after #700 had already merged into `iceberg-0.10`. Consequently, the estimator and harness changes landed only on the old topic branch. This PR cherry-picks #704's merge commit (`d3ba554`) onto `iceberg-0.10`. There are no additional code changes; the resulting tree is identical to #704's final head (`12e1512`). Includes the source-statistics-based task estimator, explicit estimation tests, distributed execution coverage, and the private-context harness with `configure_session(...)`. Validation: - `cargo test -p datafusion-distributed-iceberg --features integration --locked --test desired_task_count` — all 9 tests passed. - `cargo fmt --all -- --check` and `git diff --check` passed. - Verified exact tree equality with #704's final head. Targets `iceberg-0.10` directly; it does not depend on the separate reapplication of #703.
alexanderbianchi
added a commit
that referenced
this pull request
Sep 9, 2026
## Less test code, stronger statistics coverage Based directly on `iceberg-0.10` at `f9340ef`, including merged #715 and #716. “Before” refers to that base. Only `iceberg/tests/statistics.rs` changes, with a net reduction of 24 lines. | Test / edge case | Before | After | |---|---|---| | Row count and full-scan byte size | Separate enabled/disabled test bodies | Ordinary named tests share assertions and retain both modes | | Full-schema column statistics | Vector length only, in another pair of tests | Complete expected vector: populated metrics and unknown columns | | Reordered projection / nonconsecutive selected field IDs | Projected vector length and row count | Exact association of metrics with reordered columns using IDs 4, 1, and 5 | | Null-count aggregation across files | **None** | Exact sum of 5 | | Column-size aggregation across files | **None** | Inexact sums of 400 and 600 | | Min/max bounds and scalar types | **None** | Int32 and Int64 bounds checked with their precision; opposing extrema order exercises both updating and retaining bounds | | One file missing a column's null count | **None** | Aggregate stays `Absent`, rather than treating missing as zero | | Entirely missing column metrics | Only an all-empty-metrics fixture | Unknown column alongside columns with known metrics | | Column statistics disabled | Fixture had no metrics even when enabled | Populated fixture must return unknown column statistics when disabled | | Explicit current-snapshot selection | Separate duplicate row-total test | Fixture selects the snapshot explicitly in every matrix case | | Missing snapshot summary totals | Present | Retained | | Filter and projection/sort propagation | Present | Retained | | `COUNT(*)` skips the scan | Duplicated enabled/disabled bodies and snapshots | Both named cases retained, sharing assertions | | Explain formatting | Two identical snapshots plus weak label checks | One diagnostic snapshot; computed values verified by the matrix | ## Structure - Four ordinary named Tokio tests cover full scan / reordered projection × column statistics enabled / disabled. Two more named tests cover `COUNT(*)` scan elimination. No `test_case` macros or dependency. - Session setup uses the merged harness's `configure_session(...)`; tests never access its private context. - Native Iceberg writers create a manifest with two synthetic data-file entries and a matching manifest list. #700's `with_file` supplies the bytes to the harness. - Assertions observe query-output statistics, removing the recursive search/downcast helper and also checking projection propagation. - `insta::allow_duplicates!` is needed only for the shared, identical `COUNT(*)` inline snapshots. - The fixture is planning-only: synthetic Parquet paths are not opened. Multi-manifest merging, delete files, NDV, and verification against actual Parquet contents are not claimed here. - The genuinely historical-snapshot regression remains in #702; this PR consolidates the current-snapshot-only check from #687. ## Validation - `cargo test -p datafusion-distributed-iceberg --test statistics --locked` — all 10 cases passed - `cargo test -p datafusion-distributed-iceberg --locked` — 95 tests passed, including the doctest - `cargo test -p datafusion-distributed-iceberg --features integration --locked` — 96 tests passed, including the doctest - `cargo clippy -p datafusion-distributed-iceberg --all-targets --locked -- -D warnings` — passed, also with `--all-features` - `cargo fmt --all -- --check` - `git diff --check` - Mutation check: replacing computed column statistics with unknown values fails both enabled cases; the other eight tests pass. Mutation reverted. - Bounds mutation check: keeping the first minimum and taking the last maximum fails both enabled cases; the other eight tests pass. Mutation reverted.
alexanderbianchi
added a commit
that referenced
this pull request
Sep 9, 2026
Rebased directly onto `iceberg-0.10` after #700, #705, #715, and #716 merged. Strengthens the existing selected-snapshot statistics test rather than adding overlapping coverage. A test-local helper uses native Iceberg builders to add snapshot 42 while leaving the original taxi snapshot current. The completed metadata and `iceberg.snapshot_id = 42` are supplied through the shared harness. The test reports `Exact(42)` rows and `Exact(4242)` bytes, distinguishing the selected snapshot from the current snapshot's 175,000 rows and 4,480,382 bytes. The summary totals are deliberately synthetic. The manifest still describes the original taxi data, so this fixture tests statistics planning rather than executing a 42-row scan. This limitation is documented beside its construction. Diff against `iceberg-0.10`: one test file, 45 added lines, no production changes. Validation: - `cargo test -p datafusion-distributed-iceberg --locked --test statistics reports_statistics_for_the_selected_snapshot` — passed - `cargo fmt --all -- --check` - `git diff --check`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests can supply complete Iceberg table metadata, explicit table options, and in-memory fixture files through
IcebergTestHarness::builder().with_table_metadataaccepts native Iceberg metadata without reconstructing its IDs, schemas, or snapshot history.with_table_optionreplaces duplicated registration SQL in the selected-snapshot and invalid-snapshot-ID tests.with_file(uri, bytes)overrides a fixture file, including manifests and manifest lists. Explicit files take precedence over generated metadata; unmodified paths fall back to the checked-in taxi fixture. An empty-table registration test verifies raw metadata override behavior.build()now constructs the session and registers the table directly; there is no intermediatecreate()helper. The harness remains usable without distributed integration features.Shared fixture helpers expose only the original taxi metadata and a native metadata builder starting before the first snapshot. Snapshot recipes stay test-local. The duplicated missing-summary JSON fixture is removed.
Rebased onto
iceberg-0.10at6507bf8, including merged #687. The table-options cleanup and file-override support are separate commits. Test-case consolidation is intentionally kept in an independent PR.Validation on #700 at
bb3872e:cargo test -p datafusion-distributed-iceberg --locked— 90 tests passed, including the doctestcargo clippy -p datafusion-distributed-iceberg --all-targets --locked -- -D warningscargo fmt --all -- --checkgit diff --check