Repository navigation
[GLUTEN-12743][VL] Support generated Delta DV metadata in native scans - #13123
Draft
felipepessoto wants to merge 9 commits into
Draft
felipepessoto wants to merge 9 commits into
felipepessoto wants to merge 9 commits into
Conversation
|
Run Gluten Clickhouse CI on x86 |
2 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
2 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
Generate absolute file positions and byte-valued Delta row-index-filter results without prematurely masking rows. Preserve consumer predicates, enforce generated-key dynamic filters, and retain explicit fallback for unsupported scan shapes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Store shared filter pointers and use Filter::merge(newFilter, accumulatedFilter), matching the pinned Velox API for both newly accepted dynamic filters and preloaded-source handoff. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pass a memory-pool reference to the pinned Velox BufferedInput constructor so native Delta regression tests compile. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
FileSink creates its output exclusively. Use a path inside a scoped temporary directory instead of an already-created TempFilePath, allowing the row-group/split regression to execute. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… Delta 2.4 Move the newer file-format capability into the existing versioned DV facade and keep unsupported Delta 2.4 disabled. Make shared test sources use the portable field contract and versioned plan-capture helpers. Spark3.4/Delta2.4 and Spark3.5/Delta3.3 reactor test compilation both pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…xtures Scope Parquet row-group sizing through Hadoop configuration, resolve URI-encoded AddFile paths with Delta helpers, inspect the materialized AQE plan, and require native offload in the existing DV regression. All 27 Handoff tests and the existing DV regression pass with CI-built native libraries on Linux. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The completed eight-shard Delta run 36061115435 reported zero regressions and exactly ten now-passing cases: nine generated DV metadata cases and DELETE with DVs under id column mapping. Remove only those entries. Both previously failing gates replay successfully with fail-on-fixed still enabled, and the baseline matches the aggregate artifact exactly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep Spark metadata row-index-only scans on their existing native Parquet path, including optimized reads with useMetadataRowIndex=false. Share exact Delta field recognition, preserve explicit fallback for unsupported DV outputs and required-only fields, and retain the merged data-correctness regressions with native assertions. Cover interaction with map-key pruning from current main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
felipepessoto
force-pushed
the
felipepessoto-native-delta-dv-metadata
branch
from
October 15, 2026 17:39
53918d0 to
46ed718
Compare
|
Run Gluten Clickhouse CI on x86 |
…index shims Apply the Delta-only metadata classification consistently to the newly added Spark4.2 shim. The existing DeltaMetadataColumnSuite now passes both tests with the Spark4.2 provider instead of misclassifying the byte flag as a generic Parquet row index. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Run Gluten Clickhouse CI on x86 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Decision: not pursuing this proposal. I do not plan to take this draft forward for merge; it is retained as an exploratory implementation. Delta defaults
deletionVectors.useMetadataRowIndextotrue, and that native path already works. This proposal mainly accelerates the non-defaultfalsepath, for which #13042 already provides a JVM correctness fallback.Avoiding JVM scans and row/column conversions could benefit affected reads and DML, but no speedup has been measured. Without a demonstrated production need, the additional Substrait protocol, native reader, dynamic-filter/preloading and versioned-shim maintenance is not justified. The passing ID-mapping case also does not establish that missing Parquet field IDs or JVM interoperability have been fixed.
Partially addresses #12743 as a native follow-up to the merged JVM fallback in #13042. Rebased onto
mainat020211a4b; this does not resolve the entire issue.When
spark.databricks.delta.deletionVectors.useMetadataRowIndex=false:__delta_internal_row_indexfrom absolute file positions and__delta_internal_is_row_deletedas a byte-valued filter result. Honor contained/inverse filters, absent and empty DVs, splits, pruning and batches._metadata.row_indexpath, including optimized reads with the setting disabled. A temporary Spark row-index field alone does not select Delta metadata synthesis.Retains default metadata-row-index behavior, executor-deferred DV reads/metrics from #12836 and main's map-key pruning. Updates the merged correctness tests to require native execution for supported cases without weakening their result assertions, and adds fallback/map-pruning interaction coverage.
The nine generated-metadata baseline removals already landed in #13042; this diff retains only the previously CI-confirmed DV DELETE/id-mapping removal. No CI workflow migration is included. Deploy matching native libraries and JVM artifacts for the new Substrait column kinds.
How was this patch tested?
Validated the rebased sources in an isolated Linux container with a fresh native build against IBM Velox
dft-2026_10_14(95538c99c):DeltaDeletionVectorHandoffSuite+VeloxDeltaSuite: 74/74 on Spark 3.5/Delta 3.3 and 75/75 on Spark 4.0/Delta 4.0. Includes [GLUTEN-12743][VL] Fall back Delta scans requiring generated DV metadata #13042's metadata-row-index case, generated values, unsupported fallbacks, DML/CDF and map-key pruning.git diff --checkpassed.Spark builds use the repository Maven wrapper; Spark 4 uses JDK 17, Scala 2.13 and release 17. Fresh GitHub CI after the rebase is separate from these completed local checks. The PR remains a draft and makes no performance claim.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.95-3 (GPT-6 Astra, gpt-6-astra)