Repository navigation
[VL] Keep synthetic scan metadata out of physical file schemas - #13136
Open
felipepessoto wants to merge 5 commits into
Open
felipepessoto wants to merge 5 commits into
felipepessoto wants to merge 5 commits into
Conversation
Contributor
Author
|
/delta-test |
|
🔄 Delta Spark UT started by @felipepessoto (~2.5 h). View run |
felipepessoto
marked this pull request as ready for review
September 26, 2026 01:45
Contributor
Author
|
@philo-he @zhouyuan @zhztheplayer could you please review this PR? Thank you! |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
No unresolved approval-blocking issues were identified.
Review effort: Lite
Findings: None
What changed in this PR
Fixes metadata-only Parquet scans when synthetic metadata names collide with physical columns.
Changes:
- Separates logical metadata outputs from derived physical reader schemas.
- Preserves explicit schemas, filters, and positional mapping.
- Adds Spark/native regressions and removes 12 fixed Delta failures.
| File | Reviewed change |
|---|---|
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/GlutenFileMetadataStructSuite.scala |
Adds Spark metadata collision and mapping regressions. |
cpp/velox/tests/Substrait2VeloxPlanConversionTest.cc |
Adds native schema and execution tests. |
cpp/velox/substrait/SubstraitToVeloxPlan.cc |
Separates logical scan and physical reader schemas. |
.github/workflows/util/delta-spark-ut/known-failures.txt |
Removes verified fixed Delta cases. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Derive reader data columns from regular scan slots without changing logical output or filter binding. Preserve explicit physical schemas and existing collision fallback, with native and Spark metadata regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Disable native fixture writes and verify both empty files have readable Parquet footers, zero rows, and the conflicting INT64 field. Avoid the native writer's zero-byte output for empty input. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Serialize scan mapping modes through split metadata instead of using an ignored connector option. Assert mode forwarding and distinguish positional mapping from name mapping. Resolve the overloaded Parquet footer type lookup in the Spark empty-file test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve Spark input-file URIs before checking empty Parquet footers. Exercise a nontrivial native residual timestamp predicate while retaining string/numeric pushdown coverage; direct synthesized timestamp equality is unsupported on both baseline and fixed libraries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
All twelve original aliasing cases executed and passed in Delta run 36165204380 against merge ac916ff containing PR head 15fd9a3. The unchanged strict comparison reports zero regressions and passes after removing only these twelve stale entries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
felipepessoto
force-pushed
the
felipepessoto-metadata-schema-separation
branch
from
October 15, 2026 21:31
052015f to
de5a5cd
Compare
Contributor
Author
|
@zhztheplayer could you please review this PR? The rebase onto main is complete, and all 90 CI checks pass on de5a5cd. Thank you! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fix metadata-only scans such as
SELECT DISTINCT _metadata.file_namefailing when a Parquet file also contains a user-definedfile_name BIGINTcolumn.known-failures.txt.No Scala/Substrait protocol or cast changes. Direct timestamp-metadata equality pushdown was a pre-existing limitation in the originally tested dependency; residual timestamp filtering is covered.
How was this patch tested?
de5a5cdd9(main020211a4b), including x86, ARM, enhanced, Spark 3.4-4.2, Delta, Iceberg, and Kafka workflows.file_nameregression failed with baseline libraries and passed with this fix. Formatting and license checks pass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot 1.0.87-0 (GPT-6 Astra)