Skip to content

Head-to-head with iceberg-datafusion over the taxibench table - #1

Merged
winding-lines merged 1 commit into
mainfrom
bench-iceberg-datafusion
Oct 21, 2026
Merged

winding-lines merged 1 commit into
mainfrom
bench-iceberg-datafusion

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Runs the taxibench suite (79,478,796 trips, 24 zstd files) through DataFusion 55.2 twice: over this provider, and over apache/datafusion-iceberg's IcebergStaticTableProvider, which is iceberg-rust's DataFusion integration since apache/iceberg-rust#3228. The SQL, the thread budget and the table are the same, so only the scan differs. bench/compare.py checks every answer before printing a time, and all sixteen agree.

1 thread (mlake ÷ iceberg) 10 threads
range 0.78× – 1.62× 0.19× – 0.47×
  • 10 threads: iceberg-datafusion's scan is a single partition (UnknownPartitioning(1)), so it stays on about one core. This provider gives DataFusion one partition per ticket.
  • 1 thread: per-column decode costs about the same for both. The Mojo scan pays about 5 ms per ticket, because the stateless bridge re-reads the metadata and re-plans the manifests on every read (87 tickets per query). iceberg-rust's row filter is costly, which is why the Mojo scan still wins Q1, Q5 and Q6.

The full tables, probes and caveats are in the README section. The bench crate is separate (bench/), so iceberg-rust stays out of the example's own dependencies. It needs Rust 1.95, and the pixi floor is raised to match.

Not in CI: it needs the 1.3 GB taxibench table.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KxW5zazz1MvVNCxeZzQBRL

bench/ runs the taxibench suite through DataFusion 55.2 twice, over this
provider and over apache/datafusion-iceberg's IcebergStaticTableProvider,
with the same SQL and thread budget; compare.py checks every answer before
printing a time. All sixteen agree.

Single-threaded the Mojo scan is 0.78-1.62x iceberg's time: per-column
decode costs about the same, but each ticket pays ~5 ms of metadata reading
and manifest replanning. With 10 threads it is 0.19-0.47x, because
iceberg-datafusion's scan is a single partition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxW5zazz1MvVNCxeZzQBRL
@winding-lines
winding-lines merged commit a86dc5e into main Oct 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant