Skip to content

fix: error instead of wrong rows when a subquery repeats an equal SubqueryAlias - #27019

Open
raghavyadav01 wants to merge 2 commits into
apache:mainfrom
raghavyadav01:fix/26692-equal-subplans
Open

raghavyadav01 wants to merge 2 commits into
apache:mainfrom
raghavyadav01:fix/26692-equal-subplans

Conversation

@raghavyadav01

@raghavyadav01 raghavyadav01 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN t2 AS s) returns 3 rows instead of 9 (#26692 query 1). PullUpCorrelatedExpr keys correlated_subquery_cols_map and collected_count_expr_map by plan value, so the two equal t2 AS s share one entry and one picks up or replaces the other's correlated columns. Following #5492, these shapes now error rather than being made to work.

What changes are included in this PR?

The rewriter remembers the order in which each SubqueryAlias plan is first returned by f_up. If a node returns one that was first returned outside its own subtree, and either has correlated facts, can_pull_up is set to false and the subquery stays correlated.

Are these changes tested?

Yes. sqllogictests for q1 in EXISTS / NOT EXISTS / IN / scalar forms, the same with the plain alias first, the LIMIT case, the case where a second optimizer pass collides on the scans below the aliases, and shapes that must keep working (including two equal aliases without correlated columns of their own). The #26904 corpus case for #26692 now expects an error.

I also compared 33 shapes against DuckDB 1.5.6. Every one that changed went from wrong rows to an error, except the shapes listed below, which go from correct rows to an error.

Not covered: a LATERAL count over a correlated derived table and the same table still returns wrong rows, filed as #27566. The scalar form of that query now errors through #27244.

Are there any user-facing changes?

Queries that returned wrong rows now return an error. These shapes also error now, although they return correct rows in 55.3.0:

  • two identical correlated copies of an aliased derived table in EXISTS / NOT EXISTS, e.g. EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y)
  • the plain t2 AS s before the correlated copy in EXISTS

They are correct only because the two copies hold the same rows. Once the query reads the second copy's columns the same shape is wrong in 55.3.0: this returns 1, and the expected result is empty.

SELECT t.a FROM t WHERE EXISTS (
  SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x
  JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y ON y.b = x.b + 29
) ORDER BY 1;

AI-assisted. I reviewed every line and ran the tests above.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Nov 2, 2026
@raghavyadav01

Copy link
Copy Markdown
Contributor Author

@jayzhan211 would you have time to look at this one? Same area as #26874 (decorrelation): it turns a wrong-results case into an error when a subquery repeats the same alias.

@raghavyadav01

Copy link
Copy Markdown
Contributor Author

Heads-up on this one given alamb's note in #5492 about not breaking queries that worked in a release: this PR makes the EXISTS form of (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y error, and that returns correct rows in 55.2.0. Narrowing the guard doesn't work cleanly because both maps are keyed by plan value, so the second copy overwrites the first copy's entry before any check can compare them. The proper fix is to key them by node position instead. @jayzhan211 would you rather I do that here, or keep this PR and accept the one regression?

@jayzhan211
jayzhan211 self-requested a review November 18, 2026 13:48

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@raghavyadav01 thanks for the PR. To your question: please keep the error and don't key the maps by node position. That is what #26699 did, and alamb closed it asking for an error instead. The queries that stop working are correct only because the two copies of s hold the same rows. Once the query reads the second copy's columns, the same shape is wrong in 55.3.0: this returns 1, and the expected result is empty:

SELECT t.a FROM t WHERE EXISTS (
  SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x
  JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y ON y.b = x.b + 29
) ORDER BY 1;

Could you list the shapes that now error under "Are there any user-facing changes?" (two identical correlated copies in EXISTS / NOT EXISTS, and the plain t2 AS s first in EXISTS), with this query as the reason?


Optional cleanup of the description:

  • The Rationale opens with PullUpCorrelatedExpr internals. Opening with the symptom gets a maintainer to the point faster: "EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN t2 AS s) returns 3 rows instead of 9 (#26692 query 1); these shapes now error."
  • "Every one that changed went from wrong rows to an error, with one exception": the EXISTS forms with the plain s first and with two identical copies also go from correct rows to an error.
  • I couldn't find the issue for the "two equal derived tables without a shared alias" case. Could you link it?

let aliases_before = self.aliases_before.pop().unwrap_or(0);

let transformed = self.pull_up(plan)?;
if matches!(transformed.data, LogicalPlan::SubqueryAlias(_)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This query returns 3 on this branch, but the right answer is 1, 2, 3; 55.3.0 and main fail with a schema error instead:

CREATE TABLE t (a INT, b INT) AS VALUES (1, 1), (2, 2), (3, NULL);
CREATE TABLE t2 (a INT, b INT) AS VALUES (1, 1), (3, 30), (2, NULL);
SELECT t.a FROM t WHERE EXISTS (
  SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x
  CROSS JOIN t2 AS s WHERE s.b = 30
) ORDER BY 1;

Here's how it happens. Once the first optimizer pass keeps the subquery, push_down_filter moves both filters below their s. Now the two s differ, and the second pass collides on two equal TableScan: t2 plans instead. The join filter then reads the plain s.a, not x.a.

Dropping the SubqueryAlias condition here makes this query error. It also turns your two "without a shared alias" tests at lines 4529 and 4541 into errors, though both return correct rows in 55.3.0. Could you make this query error and add it as a query error test?

let next_order = self.alias_order.len();
match self.alias_order.get_mut(&transformed.data) {
Some((order, had_facts)) => {
if *order < aliases_before && (*had_facts || has_facts) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers the (*had_facts || has_facts) part. If it is removed, every test still passes, while two equal aliases with no correlated columns of their own start to error. Could you add this test next to the "same table twice" ones? It passes on this branch and fails without that condition:

# Two equal `t26692_2 AS s` without correlated columns of their own still
# decorrelate.
query I
SELECT t.a FROM t26692 t
WHERE EXISTS (
  SELECT 1 FROM (SELECT * FROM t26692_2 AS s) p
  CROSS JOIN (SELECT * FROM t26692_2 AS s) q
  WHERE p.a = t.a
)
ORDER BY 1;
----
1
2
3

…queryAlias

PullUpCorrelatedExpr keys correlated_subquery_cols_map and
collected_count_expr_map by plan value, so two equal SubqueryAlias
nodes in one subquery share an entry and one reads or replaces the
facts of the other. Track the order each SubqueryAlias plan is first
returned by f_up, and stop the pull up when a node returns one first
returned outside its own subtree and either has facts.

Closes apache#26692 (the wrong-results part).
Mark the apache#26904 corpus case for apache#26692 as an error. Drop the two
"without a shared alias" cases, which main now rejects through apache#27244.

Signed-off-by: Raghav Yadav <raghavmnnit@gmail.com>

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@raghavyadav01 thanks for the update, the check and the new tests look good to me.

Optional cleanup of the description: after the rebase onto #27244, the two shapes listed under "Are there any user-facing changes?" as correct in 55.3.0 already error on main without this PR, through #27244's "has the same name as an earlier column" check. So compared with main, this PR doesn't turn any query with correct rows into an error, and a maintainer reading the section with #5492 in mind shouldn't stop on it. Something like:

Queries that returned wrong rows now return an error. Compared with main, no query that returned correct rows now errors. Two shapes that return correct rows in 55.3.0 already error on main through #27244: two identical correlated copies of an aliased derived table in EXISTS / NOT EXISTS, and the plain t2 AS s before the correlated copy.

The shapes on 55.3.0, main (13efc9c) and this branch

t(a, b) = (1,1),(2,2),(3,NULL), t2(a, b) = (1,1),(3,30),(2,NULL)

Subquery in WHERE 55.3.0 main this PR
EXISTS (... (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y) 1, 2, 3 (correct) error (#27244) error
same, NOT EXISTS empty (correct) error (#27244) error
same, with FROM t, t2 AS s outside 9 rows (correct) error (#27244) error
EXISTS (... t2 AS s CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) x) 1, 2, 3 (correct) error (#27244) error
same, with FROM t, t2 AS s outside 9 rows (correct) error (#27244) error

WHERE t.a IN (
SELECT x.a FROM (SELECT * FROM t26692_2 AS s WHERE s.b = t.b) x
CROSS JOIN t26692_2 AS s
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: the same shape with IN in the select list returns wrong rows on main (false, false, true for each t.a, where every row should be true) and errors on this branch, but no test covers it. That form is rewritten to a mark join, a different path from WHERE ... IN:

Suggested change
);
);
# The same in a projection, which `decorrelate_predicate_subquery` rewrites
# to a mark join.
query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression InSubquery
SELECT t.a, t.a IN (
SELECT x.a FROM (SELECT * FROM t26692_2 AS s WHERE s.a = t.a) x
CROSS JOIN t26692_2 AS s
) FROM t26692 t, t26692_2 AS s;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants