Is your feature request related to a problem or challenge?
#25764 (which fixes #25507) stops PullUpCorrelatedExpr from pulling a correlated filter out from under the side of an outer join that gets NULL-extended, because that gave wrong results. Those subqueries now stay correlated and fail with a not-implemented error:
CREATE TABLE o(k INT) AS VALUES (1), (5);
CREATE TABLE a(id INT) AS VALUES (1), (2);
CREATE TABLE b(id INT, y INT) AS VALUES (1, 1), (2, 2);
SELECT o.k,
EXISTS (SELECT 1
FROM a LEFT JOIN (SELECT * FROM b WHERE b.y = o.k) AS b
ON a.id = b.id
WHERE b.y IS NULL) AS e
FROM o ORDER BY k;
-- expected (DuckDB): (1, true), (5, true)
-- now: This feature is not implemented: Physical plan does not support logical expression Exists(...)
The same applies to IN, WHERE [NOT] EXISTS, scalar subqueries, the left side of a RIGHT JOIN, either side of a FULL JOIN, a correlated filter nested under an inner join on the nullable side, and LATERAL subqueries (which fail with the OuterReferenceColumn not-implemented error). The sqllogictest cases added in #25764 (subquery.slt, lateral_join.slt) cover all of these.
Describe the solution you'd like
Run these queries and return the right rows. Since the filter cannot move above the join, the outer relation has to be brought down to the filter: join the distinct outer keys into the nullable input, add the outer key columns to the join condition, and group or semi-join on them above. This is the dependent join approach from Neumann and Kemper, "Unnesting Arbitrary Queries".
DuckDB 1.5.5 returns the right results for the LEFT and RIGHT cases. For the FULL case it fails with "Unsupported join type for flattening correlated subquery".
Describe alternatives you've considered
Keep the not-implemented error. It no longer returns wrong rows, but these are valid queries that DuckDB runs.
Additional context
Is your feature request related to a problem or challenge?
#25764 (which fixes #25507) stops
PullUpCorrelatedExprfrom pulling a correlated filter out from under the side of an outer join that gets NULL-extended, because that gave wrong results. Those subqueries now stay correlated and fail with a not-implemented error:The same applies to
IN,WHERE [NOT] EXISTS, scalar subqueries, the left side of aRIGHT JOIN, either side of aFULL JOIN, a correlated filter nested under an inner join on the nullable side, andLATERALsubqueries (which fail with theOuterReferenceColumnnot-implemented error). The sqllogictest cases added in #25764 (subquery.slt,lateral_join.slt) cover all of these.Describe the solution you'd like
Run these queries and return the right rows. Since the filter cannot move above the join, the outer relation has to be brought down to the filter: join the distinct outer keys into the nullable input, add the outer key columns to the join condition, and group or semi-join on them above. This is the dependent join approach from Neumann and Kemper, "Unnesting Arbitrary Queries".
DuckDB 1.5.5 returns the right results for the LEFT and RIGHT cases. For the FULL case it fails with "Unsupported join type for flattening correlated subquery".
Describe alternatives you've considered
Keep the not-implemented error. It no longer returns wrong rows, but these are valid queries that DuckDB runs.
Additional context
PullUpCorrelatedExprindatafusion/optimizer/src/decorrelate.rs; the guard added in fix: keep a correlated filter below the nullable side of an outer join #25764 is theLogicalPlan::Joinarm inf_down.Limit) and fix: keep a correlated filter below an aggregate with a grouping set #25529 (grouping sets) mark other shapes as unsupported in the same way.