Skip to content

[GLUTEN-12834][CH] Fix nullable filtering after Expand with constant columns - #12818

Merged
exmy merged 2 commits into
apache:mainfrom
exmy:codex/fix-expand-constant-null-filter
Aug 20, 2026
Merged

exmy merged 2 commits into
apache:mainfrom
exmy:codex/fix-expand-constant-null-filter

Conversation

@exmy

@exmy exmy commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

This PR fixes incorrect nullable filtering after Expand in the ClickHouse backend.

The changes:

  • Materialize constant selection columns in regular Expand.
  • Apply the same behavior to AdvancedExpand.
  • Preserve nullable types for NULL literals.
  • Add regression tests for both regular and lazy aggregate Expand paths.

Expand emits each grouping-set projection as a separate chunk.

When a selected input column originates from a SQL literal, castColumn may preserve it as a ColumnConst. In the reproducing query, the first chunk contains a constant non-null msg_type, so isNotNull(msg_type) produces a constant true filter.

The ClickHouse FilterTransform version used by the backend preserves ConstantFilterDescription between chunks. It checks the previous always_true state before recalculating the filter for the current chunk.

As a result, the following chunk, whose msg_type is a real nullable NULL, can bypass filtering and produce extra grouping-set rows.

For example:

select msg_type, os, count(*) as cnt
from (
  select 'file' as msg_type, 'Android' as os, id from range(10)
  union all
  select 'file' as msg_type, 'iOS' as os, id from range(10)
) t
group by grouping sets ((msg_type), (os))
having msg_type is not null
order by msg_type, os, cnt

the incorrect result was:

NULL  Android  10
NULL  iOS      10
file  NULL     20

The expected result is:

file  NULL  20

Materialize constant selection columns in regular and advanced Expand to prevent stale constant filter state from leaking null rows. Preserve nullable types for null literals and add regression coverage for both Expand paths.

How was this patch tested?

add ut

Was this patch authored or co-authored using generative AI tooling?

Codex

close #12834

Materialize constant selection columns in regular and advanced Expand to prevent stale constant filter state from leaking null rows. Preserve nullable types for null literals and add regression coverage for both Expand paths.
Copilot AI lite review requested due to automatic review settings August 19, 2026 02:59
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a ClickHouse backend correctness issue where Expand could emit constant columns that caused stale constant-filter state to leak across chunks, allowing rows with nullable NULL values to bypass is not null filtering. It does so by materializing constant selection columns in both regular and advanced Expand paths, preserving nullability for NULL literals, and adding regression tests covering both regular and lazy-aggregate Expand plans.

Changes:

  • Preserve nullable output types for NULL literals in Expand projections.
  • Materialize constant selection columns produced by castColumn in both ExpandTransform and AdvancedExpandTransform.
  • Add regression tests validating grouping-sets + having ... is not null behavior for both regular and lazy aggregate Expand.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
cpp-ch/local-engine/Parser/RelParsers/ExpandRelParser.cpp Ensures NULL literals remain nullable in Expand output types to produce a real null map.
cpp-ch/local-engine/Operator/ExpandTransform.cpp Converts casted selection results to full columns when they’re constant to prevent per-chunk constant filter leakage.
cpp-ch/local-engine/Operator/AdvancedExpandStep.cpp Applies the same const-materialization behavior for AdvancedExpand.
backends-clickhouse/src/test/scala/org/apache/gluten/execution/tpch/GlutenClickHouseTPCHSaltNullParquetSuite.scala Adds regression coverage for both Expand paths and toggles the runtime flag used by the CH planner.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cpp-ch/local-engine/Parser/RelParsers/ExpandRelParser.cpp Outdated
Use the ClickHouse makeNullable helper for null literals and configure the lazy aggregate Expand tests only through SparkEnv, which is the actual configuration source.
Copilot AI review requested due to automatic review settings August 19, 2026 03:12
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@exmy exmy changed the title [CH] Fix nullable filtering after Expand with constant columns [GLUTEN-12834][CH] Fix nullable filtering after Expand with constant columns Aug 20, 2026
@exmy
exmy merged commit 3817c1d into apache:main Aug 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CH] Incorrect results for GROUPING SETS with constant columns and IS NOT NULL filter

3 participants