Repository navigation
[GLUTEN-12834][CH] Fix nullable filtering after Expand with constant columns - #12818
Conversation
Materialize constant selection columns in regular and advanced Expand to prevent stale constant filter state from leaking null rows. Preserve nullable types for null literals and add regression coverage for both Expand paths.
|
Run Gluten Clickhouse CI on x86 |
There was a problem hiding this comment.
Pull request overview
This PR fixes a ClickHouse backend correctness issue where Expand could emit constant columns that caused stale constant-filter state to leak across chunks, allowing rows with nullable NULL values to bypass is not null filtering. It does so by materializing constant selection columns in both regular and advanced Expand paths, preserving nullability for NULL literals, and adding regression tests covering both regular and lazy-aggregate Expand plans.
Changes:
- Preserve nullable output types for
NULLliterals in Expand projections. - Materialize constant selection columns produced by
castColumnin bothExpandTransformandAdvancedExpandTransform. - Add regression tests validating grouping-sets +
having ... is not nullbehavior for both regular and lazy aggregate Expand.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| cpp-ch/local-engine/Parser/RelParsers/ExpandRelParser.cpp | Ensures NULL literals remain nullable in Expand output types to produce a real null map. |
| cpp-ch/local-engine/Operator/ExpandTransform.cpp | Converts casted selection results to full columns when they’re constant to prevent per-chunk constant filter leakage. |
| cpp-ch/local-engine/Operator/AdvancedExpandStep.cpp | Applies the same const-materialization behavior for AdvancedExpand. |
| backends-clickhouse/src/test/scala/org/apache/gluten/execution/tpch/GlutenClickHouseTPCHSaltNullParquetSuite.scala | Adds regression coverage for both Expand paths and toggles the runtime flag used by the CH planner. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Use the ClickHouse makeNullable helper for null literals and configure the lazy aggregate Expand tests only through SparkEnv, which is the actual configuration source.
|
Run Gluten Clickhouse CI on x86 |
What changes are proposed in this pull request?
This PR fixes incorrect nullable filtering after Expand in the ClickHouse backend.
The changes:
Expand emits each grouping-set projection as a separate chunk.
When a selected input column originates from a SQL literal,
castColumnmay preserve it as aColumnConst. In the reproducing query, the first chunk contains a constant non-nullmsg_type, soisNotNull(msg_type)produces a constant true filter.The ClickHouse
FilterTransformversion used by the backend preservesConstantFilterDescriptionbetween chunks. It checks the previousalways_truestate before recalculating the filter for the current chunk.As a result, the following chunk, whose
msg_typeis a real nullable NULL, can bypass filtering and produce extra grouping-set rows.For example:
the incorrect result was:
The expected result is:
Materialize constant selection columns in regular and advanced Expand to prevent stale constant filter state from leaking null rows. Preserve nullable types for null literals and add regression coverage for both Expand paths.
How was this patch tested?
add ut
Was this patch authored or co-authored using generative AI tooling?
Codex
close #12834