Repository navigation
Conversation
ValueStreamDataSource::next() retried by calling itself. Each exhausted iterator, end-of-stream batch, or batch fully eliminated by a dynamic filter consumed a stack frame, so with a selective dynamic filter pushed into a value stream (valueStream.dynamicFilter.enabled), a long run of eliminated batches grew the native stack in proportion to the number of skipped batches. The retry logic is now a loop. Every branch keeps its previous semantics: an exhausted stream and a null batch both release the current iterator before retrying, a fully-filtered batch retries on the same iterator, and an empty pending-iterator queue still returns an engaged null RowVectorPtr so TableScan reads it as "current split finished" rather than connector-blocked. Adds a regression test that pushes a dynamic filter eliminating 4096 consecutive batches through a single split.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Revives #12563, which the stale bot closed without review. Rebased onto main.
With
spark.gluten.sql.columnar.backend.velox.valueStream.dynamicFilter.enabled=true(#11657, off by default),ValueStreamDataSource::next()calls itself once for every batch the dynamic filter removes entirely, so a long run of such batches grows the native stack and can overflow the executor thread. This makes the retry a loop, with no other behavior change. It's needed before the filter from #11605 can be on by default.How was this patch tested?
A new
manyConsecutiveFullyFilteredBatchescase inValueStreamDynamicFilterTest(velox_operators_test) filters out 4096 consecutive batches in one split and checks that the batch after them still comes through.Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
cc @acvictor @zhztheplayer