Skip to content

feat: support fractional desired task counts - #714

Open
WilliamK112 wants to merge 6 commits into
datafusion-contrib:mainfrom
WilliamK112:codex/fractional-task-count-562
Open

feat: support fractional desired task counts#714
WilliamK112 wants to merge 6 commits into
datafusion-contrib:mainfrom
WilliamK112:codex/fractional-task-count-562

Conversation

@WilliamK112

@WilliamK112 WilliamK112 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What this changes

  • Change TaskCountAnnotation::Desired and DesiredTaskCountEventResponse::desired to carry f64, while keeping Maximum as an integer hard cap.
  • Preserve fractional desired hints through merge and union aggregation, rounding only when a concrete task count is required.
  • Keep the convenience usize handler without adding an f64 implementation to the public handler API; fractional test and user handlers are ordinary functions.
  • Compute built-in file-scan hints with true floating-point division.
  • Treat Desired(0) as a valid empty-stage outcome, including safe zero-task union allocation, traversal, specialization, and execution.
  • Return a neutral Desired(0.0) consumer hint from dynamic planning so sub-one fractional hints are not masked or inflated.
  • Render non-integer desired annotations to two decimal places while preserving whole-number output.
  • Update built-in handlers, examples, public documentation, and the 5.0 upgrade guide.

The focused union regressions prove both sides of the behavior: fractional child hints are added before resolution, and two zero-valued leaves produce a valid empty stage.

This revision reapplies the maintainer-reviewed suggestions from #617. The four corresponding commits retain Nishchaya Sharma's original authorship.

Closes #562.

Validation

  • cargo fmt --all -- --check
  • cargo test --lib — 305 passed, 1 ignored
  • focused network-boundary planner tests — 25 passed
  • cargo clippy --all-targets --all-features -- -D warnings
  • git diff --check

AI assistance disclosure

I used OpenAI Codex to help inspect the planner paths, implement the change, reconcile the prior review suggestions with current main, and run validation. I reviewed the final diff and test results.

@gabotechs

Copy link
Copy Markdown
Collaborator

I see this is committing the same mistakes than #617. I recommend to go there, give it a look, and re-apply the same suggestions here.

WilliamK112 and others added 6 commits September 8, 2026 14:37
Keep Desired Debug to two decimal places for non-integers, compute
default file-scan task counts with true f64 division, and add a union
regression for 0.0 leaf hints. Two zero hints still sum before rounding
and are then rejected as a zero-task ChildrenIsolatorUnionExec.
Empty union children can sum to Desired(0). That is a valid empty
stage, not an internal planning error. Return Desired(0.0) from the
dynamic planner so UNION children can sum small fractional hints
instead of being masked by 1.0.
v4.0.0 is now released, so the Desired(f64) public break belongs in the next major upgrade guide rather than the historical 3.0.0 notes.
@WilliamK112
WilliamK112 force-pushed the codex/fractional-task-count-562 branch from f7930b3 to d79f887 Compare September 8, 2026 19:48
@WilliamK112

Copy link
Copy Markdown
Contributor Author

Thanks for pointing me to #617. I went through every maintainer thread there and reapplied the resulting design fixes on top of current main:

  • removed the public DesiredTaskCountHandler for f64 convenience impl and the direct ceil() unit test
  • switched the cache sentinel to f64::MAX
  • made file-scan hints use true floating-point division
  • limited fractional Debug output to two decimals
  • made the dynamic consumer hint neutral with Desired(0.0)
  • made Desired(0) a valid empty stage, including union allocation/traversal/execution safety
  • added end-to-end union coverage for both fractional aggregation and two zero-valued leaves
  • moved the breaking-change note to the 5.0 upgrade guide

I also rebased onto current main. The reviewed #617 follow-up commits retain Nishchaya Sharma's authorship, and the small final test reconciliation accounts for the newer cardinality-factor behavior from #713.

Fresh local validation passed: 305 library tests (1 ignored), all 25 network-boundary planner tests, fmt, all-target/all-feature clippy with warnings denied, and whitespace checks. Hosted CI is now running on d79f887.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow DesiredTaskCountHandler to return f64 as desired task counts

3 participants