Skip to content

feat: Add plan compression and CLI decode support - #586

Open
Rich-T-kid wants to merge 1 commit into
datafusion-contrib:mainfrom
Rich-T-kid:rich-T-kid/add-plan-compression
Open

feat: Add plan compression and CLI decode support#586
Rich-T-kid wants to merge 1 commit into
datafusion-contrib:mainfrom
Rich-T-kid:rich-T-kid/add-plan-compression

Conversation

@Rich-T-kid

@Rich-T-kid Rich-T-kid commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Closes #523

Why?

What Changed

  • Added protobuf-backed PlanSnapshot / PlanNodeSnapshot encode and decode support.
  • Captures each plan node’s display name and runtime metrics using the existing metrics protobuf conversion path.
  • Makes distributed EXPLAIN ANALYZE return a base64-encoded snapshot when the grpc feature is enabled.
  • Exposes DistributedExec::decode_plan_snapshot.
  • Adds datafusion-distributed-console --encoded-plan for one-off decoding.
  • Documents the encoded-plan workflow in the metrics guide.

Test

  • updated tests/explain_analyze.rs to expect the new output format

Comment thread src/coordinator/distributed.rs Outdated
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/add-plan-compression branch from e71218d to 0bb1777 Compare September 5, 2026 03:15
@Rich-T-kid
Rich-T-kid marked this pull request as draft September 5, 2026 03:15
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/add-plan-compression branch 6 times, most recently from 706f46e to 22168fa Compare September 5, 2026 23:12

@Rich-T-kid Rich-T-kid left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gabotechs this PR is ready for review when you get a chance 🚀

Comment thread tests/explain_analyze.rs
assert_contains!(&formatted, "NetworkShuffleExec");
assert_contains!(&formatted, "metrics=[output_rows=");
assert_not_contains!(&formatted, "metrics=[output_rows={");
assert_contains!(&formatted, "output_rows"); // e.g. output_rows{partition=0}=2

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: the output format for output_rows comes from how datafusion Metric implements display so I don't think we can do anything about it here

@Rich-T-kid
Rich-T-kid marked this pull request as ready for review September 5, 2026 23:16
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/add-plan-compression branch 6 times, most recently from fdd621c to bd95a6e Compare September 6, 2026 01:23
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/add-plan-compression branch from bd95a6e to 9c7061f Compare September 6, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a standard compressed serialized representation of plans with metrics

2 participants