Skip to content

Hold the plan in the bridge instead of replanning every read - #2

Merged
winding-lines merged 1 commit into
mainfrom
plan-handle
Oct 22, 2026
Merged

winding-lines merged 1 commit into
mainfrom
plan-handle

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Follow-up to #1. That head-to-head found the Mojo scan paying about 5 ms per ticket, because the bridge kept nothing between calls and every read re-planned the table to find its own byte range.

What changed

  • mlake_plan(dir, columns, filter, split) plans once and returns a handle. It holds the file scan tasks (data file and range, delete files, residual) and the schema, column ids, specs and name mapping a read needs.
  • mlake_plan_read(plan, i) reads task i with iceberg.mojo's public read_data_file, so iceberg.mojo itself doesn't change. A read only looks at the plan, so DataFusion's threads read one plan concurrently.
  • On the Rust side, Plan owns the handle (Send + Sync, freed on drop), and MlakeExec holds an Arc<Plan>.
  • mlake_read_split, mlake_plan_ticket and the string tickets are removed. "Tickets" is renamed "tasks" throughout, including the EXPLAIN line.
  • The module docstring said the bridge had to keep nothing between calls, because Mojo couldn't hold a type with a destructor on the heap. Mojo 1.1 can, with unsafe_alloc plus unsafe_write, so the docstring now explains the handle instead.

Results (taxibench, M4, both legs re-run)

All 16 answers agree.

1 thread, before 1 thread, now 10 threads, before 10 threads, now
mlake ÷ iceberg 0.78–1.62× 0.63–1.34× 0.19–0.47× 0.16–0.46×
sum(trip_distance) probe 670 ms 454 ms

Every query that reads the whole table got 160–220 ms faster single-threaded. The Mojo scan now wins 5 of 8 queries on one thread, up from 4.

Still open

About 2.5 ms per task remains, and it's inside read_data_file. Calling it directly on the 87 tasks takes 433 ms, against 275 ms for the same rows read as 24 whole files. What I ruled out:

  • Planning: it's gone now.
  • The footer: 12 ms for all 87 tasks.
  • Over-fetching: a task reads only the row groups it owns.

That leaves a per-call cost somewhere in iceberg.mojo or parquet.mojo. The README says this.

Testing

  • pixi run test: 4 unit tests and 6 oracle tests pass.
  • cargo clippy --all-targets is clean.
  • cargo fmt and mojo format applied.
  • pixi run bench re-run on both legs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KxW5zazz1MvVNCxeZzQBRL

mlake_plan now takes the column selection with the filter, plans once,
and returns a handle to the file scan tasks plus the schema, specs and
name mapping a read needs. mlake_plan_read reads one task from it through
iceberg.mojo's read_data_file; reads only look at the plan, so
DataFusion's threads share one. Rust owns the handle as a Plan and frees
it on drop.

The bridge used to keep nothing between calls, on the grounds that Mojo
could not hold an object with a destructor on the heap. Mojo 1.1 can
(unsafe_alloc + unsafe_write). Without that, every read reopened the
metadata and replanned the manifests to find its own ticket again, about
3 ms per read.

taxibench, 1 thread: 0.63-1.34x iceberg-datafusion (was 0.78-1.62x);
10 threads: 0.16-0.46x (was 0.19-0.47x). All 16 answers agree. About
2.5 ms per task remains, inside read_data_file; the README records
what it is not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxW5zazz1MvVNCxeZzQBRL
@winding-lines
winding-lines merged commit 2667087 into main Oct 22, 2026
2 checks passed
@winding-lines
winding-lines deleted the plan-handle branch October 22, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant