Repository navigation
fix: error instead of wrong rows when a subquery repeats an equal SubqueryAlias - #27019
raghavyadav01 wants to merge 2 commits into
Conversation
|
@jayzhan211 would you have time to look at this one? Same area as #26874 (decorrelation): it turns a wrong-results case into an error when a subquery repeats the same alias. |
|
Heads-up on this one given alamb's note in #5492 about not breaking queries that worked in a release: this PR makes the EXISTS form of |
jayzhan211
left a comment
There was a problem hiding this comment.
@raghavyadav01 thanks for the PR. To your question: please keep the error and don't key the maps by node position. That is what #26699 did, and alamb closed it asking for an error instead. The queries that stop working are correct only because the two copies of s hold the same rows. Once the query reads the second copy's columns, the same shape is wrong in 55.3.0: this returns 1, and the expected result is empty:
SELECT t.a FROM t WHERE EXISTS (
SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x
JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y ON y.b = x.b + 29
) ORDER BY 1;Could you list the shapes that now error under "Are there any user-facing changes?" (two identical correlated copies in EXISTS / NOT EXISTS, and the plain t2 AS s first in EXISTS), with this query as the reason?
Optional cleanup of the description:
- The Rationale opens with
PullUpCorrelatedExprinternals. Opening with the symptom gets a maintainer to the point faster: "EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN t2 AS s)returns 3 rows instead of 9 (#26692 query 1); these shapes now error." - "Every one that changed went from wrong rows to an error, with one exception": the EXISTS forms with the plain
sfirst and with two identical copies also go from correct rows to an error. - I couldn't find the issue for the "two equal derived tables without a shared alias" case. Could you link it?
| let aliases_before = self.aliases_before.pop().unwrap_or(0); | ||
|
|
||
| let transformed = self.pull_up(plan)?; | ||
| if matches!(transformed.data, LogicalPlan::SubqueryAlias(_)) { |
There was a problem hiding this comment.
This query returns 3 on this branch, but the right answer is 1, 2, 3; 55.3.0 and main fail with a schema error instead:
CREATE TABLE t (a INT, b INT) AS VALUES (1, 1), (2, 2), (3, NULL);
CREATE TABLE t2 (a INT, b INT) AS VALUES (1, 1), (3, 30), (2, NULL);
SELECT t.a FROM t WHERE EXISTS (
SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x
CROSS JOIN t2 AS s WHERE s.b = 30
) ORDER BY 1;Here's how it happens. Once the first optimizer pass keeps the subquery, push_down_filter moves both filters below their s. Now the two s differ, and the second pass collides on two equal TableScan: t2 plans instead. The join filter then reads the plain s.a, not x.a.
Dropping the SubqueryAlias condition here makes this query error. It also turns your two "without a shared alias" tests at lines 4529 and 4541 into errors, though both return correct rows in 55.3.0. Could you make this query error and add it as a query error test?
| let next_order = self.alias_order.len(); | ||
| match self.alias_order.get_mut(&transformed.data) { | ||
| Some((order, had_facts)) => { | ||
| if *order < aliases_before && (*had_facts || has_facts) { |
There was a problem hiding this comment.
No test covers the (*had_facts || has_facts) part. If it is removed, every test still passes, while two equal aliases with no correlated columns of their own start to error. Could you add this test next to the "same table twice" ones? It passes on this branch and fails without that condition:
# Two equal `t26692_2 AS s` without correlated columns of their own still
# decorrelate.
query I
SELECT t.a FROM t26692 t
WHERE EXISTS (
SELECT 1 FROM (SELECT * FROM t26692_2 AS s) p
CROSS JOIN (SELECT * FROM t26692_2 AS s) q
WHERE p.a = t.a
)
ORDER BY 1;
----
1
2
3…queryAlias PullUpCorrelatedExpr keys correlated_subquery_cols_map and collected_count_expr_map by plan value, so two equal SubqueryAlias nodes in one subquery share an entry and one reads or replaces the facts of the other. Track the order each SubqueryAlias plan is first returned by f_up, and stop the pull up when a node returns one first returned outside its own subtree and either has facts. Closes apache#26692 (the wrong-results part).
Mark the apache#26904 corpus case for apache#26692 as an error. Drop the two "without a shared alias" cases, which main now rejects through apache#27244. Signed-off-by: Raghav Yadav <raghavmnnit@gmail.com>
7a08e91 to
041a23c
Compare
jayzhan211
left a comment
There was a problem hiding this comment.
@raghavyadav01 thanks for the update, the check and the new tests look good to me.
Optional cleanup of the description: after the rebase onto #27244, the two shapes listed under "Are there any user-facing changes?" as correct in 55.3.0 already error on main without this PR, through #27244's "has the same name as an earlier column" check. So compared with main, this PR doesn't turn any query with correct rows into an error, and a maintainer reading the section with #5492 in mind shouldn't stop on it. Something like:
Queries that returned wrong rows now return an error. Compared with main, no query that returned correct rows now errors. Two shapes that return correct rows in 55.3.0 already error on main through #27244: two identical correlated copies of an aliased derived table in
EXISTS/NOT EXISTS, and the plaint2 AS sbefore the correlated copy.
The shapes on 55.3.0, main (13efc9c) and this branch
t(a, b) = (1,1),(2,2),(3,NULL), t2(a, b) = (1,1),(3,30),(2,NULL)
Subquery in WHERE |
55.3.0 | main | this PR |
|---|---|---|---|
EXISTS (... (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y) |
1, 2, 3 (correct) | error (#27244) | error |
same, NOT EXISTS |
empty (correct) | error (#27244) | error |
same, with FROM t, t2 AS s outside |
9 rows (correct) | error (#27244) | error |
EXISTS (... t2 AS s CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) x) |
1, 2, 3 (correct) | error (#27244) | error |
same, with FROM t, t2 AS s outside |
9 rows (correct) | error (#27244) | error |
| WHERE t.a IN ( | ||
| SELECT x.a FROM (SELECT * FROM t26692_2 AS s WHERE s.b = t.b) x | ||
| CROSS JOIN t26692_2 AS s | ||
| ); |
There was a problem hiding this comment.
Optional: the same shape with IN in the select list returns wrong rows on main (false, false, true for each t.a, where every row should be true) and errors on this branch, but no test covers it. That form is rewritten to a mark join, a different path from WHERE ... IN:
| ); | |
| ); | |
| # The same in a projection, which `decorrelate_predicate_subquery` rewrites | |
| # to a mark join. | |
| query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression InSubquery | |
| SELECT t.a, t.a IN ( | |
| SELECT x.a FROM (SELECT * FROM t26692_2 AS s WHERE s.a = t.a) x | |
| CROSS JOIN t26692_2 AS s | |
| ) FROM t26692 t, t26692_2 AS s; |
Which issue does this PR close?
Rationale for this change
EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN t2 AS s)returns 3 rows instead of 9 (#26692 query 1).PullUpCorrelatedExprkeyscorrelated_subquery_cols_mapandcollected_count_expr_mapby plan value, so the two equalt2 AS sshare one entry and one picks up or replaces the other's correlated columns. Following #5492, these shapes now error rather than being made to work.What changes are included in this PR?
The rewriter remembers the order in which each
SubqueryAliasplan is first returned byf_up. If a node returns one that was first returned outside its own subtree, and either has correlated facts,can_pull_upis set to false and the subquery stays correlated.Are these changes tested?
Yes. sqllogictests for q1 in EXISTS / NOT EXISTS / IN / scalar forms, the same with the plain alias first, the
LIMITcase, the case where a second optimizer pass collides on the scans below the aliases, and shapes that must keep working (including two equal aliases without correlated columns of their own). The #26904 corpus case for #26692 now expects an error.I also compared 33 shapes against DuckDB 1.5.6. Every one that changed went from wrong rows to an error, except the shapes listed below, which go from correct rows to an error.
Not covered: a
LATERALcount over a correlated derived table and the same table still returns wrong rows, filed as #27566. The scalar form of that query now errors through #27244.Are there any user-facing changes?
Queries that returned wrong rows now return an error. These shapes also error now, although they return correct rows in 55.3.0:
EXISTS/NOT EXISTS, e.g.EXISTS (SELECT 1 FROM (SELECT * FROM t2 AS s WHERE s.a = t.a) x CROSS JOIN (SELECT * FROM t2 AS s WHERE s.a = t.a) y)t2 AS sbefore the correlated copy inEXISTSThey are correct only because the two copies hold the same rows. Once the query reads the second copy's columns the same shape is wrong in 55.3.0: this returns
1, and the expected result is empty.AI-assisted. I reviewed every line and ran the tests above.