Repository navigation
fix: keep a sub-expression as the canonical expression of its equivalence class - #26967
Conversation
jayzhan211
left a comment
There was a problem hiding this comment.
@abp002 thanks for tracking this down! The fix works, and on current main it also fixes the window case and #27287. I left three small asks inline: run the promotion in EquivalenceClass::push/extend so classes built through EquivalenceGroup::new or extend get it too, and add tests for the window shape and for the merge/push arms of add_equal_conditions.
| /// of this class. Otherwise, with `a + b` as the canonical expression of a | ||
| /// class that also holds `a`, normalization would replace the `a` inside | ||
| /// `a + b` and turn `a + b` into `(a + b) + b`. | ||
| fn promote_sub_expressions(&mut self) { |
There was a problem hiding this comment.
@abp002 thanks for the fix! Classes built through EquivalenceGroup::new or merged by EquivalenceGroup::extend (via bridge_classes) skip this promotion, so EquivalenceGroup::new([EquivalenceClass::new([a + b, a])]) still normalizes a to (a + b) + b. Both are public. Could you run the promotion inside EquivalenceClass::push and extend instead? Then every way of building a class gets it, and the four calls in add_equal_conditions go away:
self.exprs.insert(expr);
+ self.promote_sub_expressions();
}
@@
pub fn extend(&mut self, other: Self) {
self.exprs.extend(other.exprs);
+ self.promote_sub_expressions();
@@
self.classes[first_idx].extend(other_class);
- self.classes[first_idx].promote_sub_expressions();
@@
self.classes[group_idx].push(right);
- self.classes[group_idx].promote_sub_expressions();
@@
self.classes[group_idx].push(left);
- self.classes[group_idx].promote_sub_expressions();
@@
- let mut class = EquivalenceClass::new([left, right]);
- class.promote_sub_expressions();
+ let class = EquivalenceClass::new([left, right]);With this change on current main, the physical-expr unit tests and the full sqllogictest suite pass.
Unit test for both paths (fails on the current PR, passes with the change)
#[test]
fn test_sub_expression_is_canonical_on_every_path() -> Result<()> {
let schema = abc_schema();
let (a, b, c) = (col("a", &schema)?, col("b", &schema)?, col("c", &schema)?);
let a_plus_b = binary(Arc::clone(&a), Operator::Plus, b, &schema)?;
// A class built directly, with the larger expression first
let group = EquivalenceGroup::new([EquivalenceClass::new([
Arc::clone(&a_plus_b),
Arc::clone(&a),
])]);
assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a));
// Two classes bridged by `extend` through the shared `c`
let mut group = EquivalenceGroup::default();
group.add_equal_conditions(Arc::clone(&a_plus_b), Arc::clone(&c));
let mut other = EquivalenceGroup::default();
other.add_equal_conditions(c, Arc::clone(&a));
group.extend(other);
assert!(group.normalize_expr(a_plus_b).eq(&a));
Ok(())
}| statement ok | ||
| DROP TABLE eq_in; |
There was a problem hiding this comment.
@abp002 the same bug also breaks a window on main, without a sort-merge join: PARTITION BY a + b ORDER BY a under WHERE a + b = a fails at runtime with Expects PARTITION BY expression to be ordered. This PR fixes it, but nothing tests that path. Could you add it?
| statement ok | |
| DROP TABLE eq_in; | |
| statement ok | |
| DROP TABLE eq_in; | |
| # With a + b = a, the window's PARTITION BY a + b is the same key as the | |
| # ORDER BY a below it | |
| statement ok | |
| CREATE TABLE eq_sub (a INT, b INT) AS VALUES (1, 0), (2, 3), (1, 0), (5, 0); | |
| query II | |
| SELECT a, sum(b) OVER (PARTITION BY a + b ORDER BY a) FROM eq_sub WHERE a + b = a ORDER BY a; | |
| ---- | |
| 1 0 | |
| 1 0 | |
| 5 0 | |
| statement ok | |
| DROP TABLE eq_sub; |
| 1 0 1 0 | ||
|
|
||
| statement ok | ||
| DROP TABLE smj_self_eq; |
There was a problem hiding this comment.
Removing the promotion after the merge (extend) or after push(right) in add_equal_conditions still passes the tests here, because the issue query only reaches the new-class arm. These two queries reach the other arms. Both fail with SanityCheckPlan on main, and each catches one of those removals:
| DROP TABLE smj_self_eq; | |
| DROP TABLE smj_self_eq; | |
| statement ok | |
| CREATE TABLE smj_self_eq_cd (a INT, b INT, c INT, d INT) AS VALUES (1, 0, 1, 1), (2, 3, 5, 2); | |
| # a joins the class of a + b after the class exists | |
| query IIII | |
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | |
| ---- | |
| 1 0 1 0 | |
| # a reaches the class of a + b when two classes merge | |
| query IIII | |
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | |
| ---- | |
| 1 0 1 0 | |
| statement ok | |
| DROP TABLE smj_self_eq_cd; |
| statement ok | ||
| DROP TABLE eq_t; | ||
|
|
||
| # The two IN lists hold the same values in a different order, so they compare |
There was a problem hiding this comment.
Optional cleanup: this test doesn't reach the loop bound in promote_sub_expressions. With the bound replaced by loop, the query still finishes, because neither IN list is inside the other. A promoted expression is always strictly inside the previous canonical expression, so it can't cycle. I'd drop this test, or change its comment so it doesn't claim to cover termination.
…dow and the other arms
7b2d9b2 to
a2bc6db
Compare
|
Thanks for the review, and for the tests. I moved the promotion into I also dropped the Rebased on main. |
jayzhan211
left a comment
There was a problem hiding this comment.
@abp002 thanks for the updates, all four look good. The new queries fail on main and pass here, and the full sqllogictest suite passes. I left one optional test suggestion inline.
| fn promote_sub_expressions(&mut self) { | ||
| // Each promoted expression is inside the previous canonical | ||
| // expression, so there are at most as many rounds as members. | ||
| for _ in 0..self.exprs.len() { |
There was a problem hiding this comment.
Optional: no test needs a second round of this loop. With for _ in 0..1 every test still passes. A second round happens when extend brings in a + b and a behind an unrelated canonical expression. This test passes here and fails with one round:
#[test]
fn test_sub_expression_promotion_takes_several_rounds() -> Result<()> {
let schema = abc_schema();
let (a, b, c) = (col("a", &schema)?, col("b", &schema)?, col("c", &schema)?);
let a_plus_b = binary(Arc::clone(&a), Operator::Plus, Arc::clone(&b), &schema)?;
let a_plus_b_plus_c = binary(
Arc::clone(&a_plus_b),
Operator::Plus,
Arc::clone(&c),
&schema,
)?;
// `b * c` is canonical in `other`, so `a + b` stays ahead of `a` there
let b_times_c = binary(b, Operator::Multiply, c, &schema)?;
let other = EquivalenceClass::new([b_times_c, a_plus_b, Arc::clone(&a)]);
let mut class = EquivalenceClass::new([a_plus_b_plus_c]);
class.extend(other);
assert!(class.canonical_expr().unwrap().eq(&a));
Ok(())
}apache#26967 fixes apache#27287 in EquivalenceClass, so the scan no longer needs to keep the literal out of the class. The scan change also added sorts to plans that rely on equalities with no column on either side. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
adriangb
left a comment
There was a problem hiding this comment.
Thanks, this fixes the case in the issue. The same failure still happens when two classes refer to each other:
set datafusion.optimizer.prefer_hash_join = false;
CREATE TABLE t (a BIGINT, b BIGINT, c BIGINT) AS VALUES (1, 0, 1), (2, 3, 5);
SELECT * FROM t x JOIN t y ON x.a = y.a AND x.b = y.b
WHERE x.a + x.b = x.c AND x.c + x.b = x.a;SanityCheckPlan: ... does not satisfy order requirements: [a@0 ASC, b@1 ASC].
Child-0 order: [[c@2 + b@1 ASC, b@1 ASC]]
With prefer_hash_join = true it returns 1 0 1 1 0 1.
The classes are {a + b, c} and {c + b, a}. Neither canonical expression contains a member of its own class, so promote_sub_expressions does nothing. But each one contains the other class's member, so normalize_expr is still not idempotent:
a -> c + b -> (a + b) + b -> ...
The general rule is: no canonical expression may contain a member of any class. This PR enforces it inside one class only.
I am fine merging this as a step forward. Could we open a follow-up for the general fix? Some prior art:
| System | Representative | Rewrites members by other members? |
|---|---|---|
Trino EqualityInference |
cheapest: fewest symbols, then smallest tree | no, it unions derived rewrites |
PostgreSQL equivclass.c |
none: pathkeys point at the class | no |
CockroachDB equiv_set.go |
lowest column ID (columns only) | n/a |
A possible shape for the follow-up:
- Pick the canonical expression by cost, like Trino: fewest columns, then fewest nodes, then a stable tiebreak. That makes
abeata + bwithout a special promotion pass. - When a class is added or merged, check the canonical expressions across classes. If a canonical expression contains a member of another class (directly or through a cycle), choose a different canonical expression, or skip that rewrite.
- Add
debug_assert_eq!(normalize(normalize(e)), normalize(e))innormalize_expr, so the next case fails in tests instead of inSanityCheckPlan.
Two small things:
- The PR description mentions a test with two
INlists in a different order. I cannot find it in the diff. - Could the slt tests include an
EXPLAIN? Today they only show that the query no longer errors.
| } | ||
| } | ||
| self.exprs.insert(expr); | ||
| self.promote_sub_expressions(); |
There was a problem hiding this comment.
EquivalenceClass::new calls push per member, so building a class of n members runs n promotion passes. new could insert everything and promote once, as extend does.
| let children = canonical.children(); | ||
| let Some(idx) = self.exprs.iter().skip(1).position(|expr| { | ||
| // Only the children are searched: an expression equal to the | ||
| // canonical expression itself is not inside it. The unwrap is | ||
| // safe because the closure always returns `Ok`. | ||
| children | ||
| .iter() | ||
| .any(|child| child.exists(|node| Ok(node == expr)).unwrap()) | ||
| }) else { | ||
| return; | ||
| }; | ||
| // `idx` skips the canonical expression, so the expression to | ||
| // promote is at `idx + 1`. | ||
| self.exprs.move_index(idx + 1, 0); |
There was a problem hiding this comment.
Keeping the real index removes the + 1 and its comment:
| let children = canonical.children(); | |
| let Some(idx) = self.exprs.iter().skip(1).position(|expr| { | |
| // Only the children are searched: an expression equal to the | |
| // canonical expression itself is not inside it. The unwrap is | |
| // safe because the closure always returns `Ok`. | |
| children | |
| .iter() | |
| .any(|child| child.exists(|node| Ok(node == expr)).unwrap()) | |
| }) else { | |
| return; | |
| }; | |
| // `idx` skips the canonical expression, so the expression to | |
| // promote is at `idx + 1`. | |
| self.exprs.move_index(idx + 1, 0); | |
| let children = canonical.children(); | |
| // Only the children are searched: an expression equal to the | |
| // canonical expression itself is not inside it. | |
| let Some(idx) = self.exprs.iter().enumerate().skip(1).find_map(|(idx, expr)| { | |
| children | |
| .iter() | |
| .any(|child| child.exists(|node| Ok(node == expr)).unwrap()) | |
| .then_some(idx) | |
| }) else { | |
| return; | |
| }; | |
| self.exprs.move_index(idx, 0); |
| Arc::clone(&a_plus_b), | ||
| Arc::clone(&a), | ||
| ])]); | ||
| assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a)); |
There was a problem hiding this comment.
assert_eq! prints both sides on failure:
| assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a)); | |
| assert_eq!(&group.normalize_expr(Arc::clone(&a_plus_b)), &a); |
(Same for L1790 and L1811.)
| statement ok | ||
| RESET datafusion.execution.batch_size; | ||
|
|
||
| # Regression test for #26623: a filter that equates an expression with one of |
There was a problem hiding this comment.
| # Regression test for #26623: a filter that equates an expression with one of | |
| # Regression test for https://github.com/apache/datafusion/issues/26623: a filter that equates an expression with one of |
| statement ok | ||
| CREATE TABLE eq_t (a BIGINT, b BIGINT, c BIGINT) AS VALUES (1, 1, 2), (3, 3, 4), (5, 5, 0), (2, 2, 3); | ||
|
|
||
| # The filter puts a and b in one class, and c and a + 1 in another. The inner |
There was a problem hiding this comment.
This query has no sub-expression in a class. It guards against a fix inside normalize_expr. Could the comment say so?
|
Thanks for the review! Pushed:
I opened #27537 for the two classes that refer to each other, with your query and the three steps. |
|
Thanks! @jayzhan211 are you happy to merge this? |
| # a joins the class of a + b after the class exists | ||
| query TT | ||
| EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | ||
| ---- | ||
| logical_plan | ||
| 01)Inner Join: x.a = y.a, x.b = y.b | ||
| 02)--SubqueryAlias: x | ||
| 03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b | ||
| 04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a | ||
| 05)--------TableScan: smj_self_eq_cd projection=[a, b, c] | ||
| 06)--SubqueryAlias: y | ||
| 07)----TableScan: smj_self_eq_cd projection=[a, b] | ||
| physical_plan | ||
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | ||
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | ||
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | ||
| 04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0, projection=[a@0, b@1] | ||
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | ||
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | ||
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | ||
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | ||
|
|
||
| query IIII | ||
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | ||
| ---- | ||
| 1 0 1 0 | ||
|
|
||
| # a reaches the class of a + b when two classes merge | ||
| query TT | ||
| EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | ||
| ---- | ||
| logical_plan | ||
| 01)Inner Join: x.a = y.a, x.b = y.b | ||
| 02)--SubqueryAlias: x | ||
| 03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b | ||
| 04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c | ||
| 05)--------TableScan: smj_self_eq_cd projection=[a, b, c, d] | ||
| 06)--SubqueryAlias: y | ||
| 07)----TableScan: smj_self_eq_cd projection=[a, b] | ||
| physical_plan | ||
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | ||
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | ||
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | ||
| 04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2, projection=[a@0, b@1] | ||
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | ||
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | ||
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | ||
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | ||
|
|
||
| query IIII | ||
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | ||
| ---- | ||
| 1 0 1 0 |
There was a problem hiding this comment.
@abp002 thanks for the updates! With new no longer calling push, deleting self.promote_sub_expressions() from push now passes every test. These two queries select only x.a, x.b, so the filter gets projection=[a@0, b@1], and that projection rebuilds the class through EquivalenceClass::new, which promotes anyway. With SELECT * there is no projection, so removing the promotion from push fails the first query, removing it from extend fails the second, and both still fail on main:
| # a joins the class of a + b after the class exists | |
| query TT | |
| EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | |
| ---- | |
| logical_plan | |
| 01)Inner Join: x.a = y.a, x.b = y.b | |
| 02)--SubqueryAlias: x | |
| 03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b | |
| 04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a | |
| 05)--------TableScan: smj_self_eq_cd projection=[a, b, c] | |
| 06)--SubqueryAlias: y | |
| 07)----TableScan: smj_self_eq_cd projection=[a, b] | |
| physical_plan | |
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | |
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0, projection=[a@0, b@1] | |
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | |
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | |
| query IIII | |
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | |
| ---- | |
| 1 0 1 0 | |
| # a reaches the class of a + b when two classes merge | |
| query TT | |
| EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | |
| ---- | |
| logical_plan | |
| 01)Inner Join: x.a = y.a, x.b = y.b | |
| 02)--SubqueryAlias: x | |
| 03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b | |
| 04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c | |
| 05)--------TableScan: smj_self_eq_cd projection=[a, b, c, d] | |
| 06)--SubqueryAlias: y | |
| 07)----TableScan: smj_self_eq_cd projection=[a, b] | |
| physical_plan | |
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | |
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2, projection=[a@0, b@1] | |
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | |
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | |
| query IIII | |
| SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | |
| ---- | |
| 1 0 1 0 | |
| # Both queries keep every column of x: a projection above the filter would | |
| # rebuild the class through EquivalenceClass::new, which promotes on its own | |
| # a joins the class of a + b after the class exists | |
| query TT | |
| EXPLAIN SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | |
| ---- | |
| logical_plan | |
| 01)Inner Join: x.a = y.a, x.b = y.b | |
| 02)--SubqueryAlias: x | |
| 03)----Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a | |
| 04)------TableScan: smj_self_eq_cd projection=[a, b, c, d] | |
| 05)--SubqueryAlias: y | |
| 06)----TableScan: smj_self_eq_cd projection=[a, b, c, d] | |
| physical_plan | |
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | |
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0 | |
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | |
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | |
| query IIIIIIII | |
| SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a; | |
| ---- | |
| 1 0 1 1 1 0 1 1 | |
| # a reaches the class of a + b when two classes merge | |
| query TT | |
| EXPLAIN SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | |
| ---- | |
| logical_plan | |
| 01)Inner Join: x.a = y.a, x.b = y.b | |
| 02)--SubqueryAlias: x | |
| 03)----Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c | |
| 04)------TableScan: smj_self_eq_cd projection=[a, b, c, d] | |
| 05)--SubqueryAlias: y | |
| 06)----TableScan: smj_self_eq_cd projection=[a, b, c, d] | |
| physical_plan | |
| 01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)] | |
| 02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2 | |
| 05)--------DataSourceExec: partitions=1, partition_sizes=[1] | |
| 06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true] | |
| 07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1 | |
| 08)------DataSourceExec: partitions=1, partition_sizes=[1] | |
| query IIIIIIII | |
| SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c; | |
| ---- | |
| 1 0 1 1 1 0 1 1 |
|
Thanks, @jayzhan211! Applied the suggestion: both queries now use |
Which issue does this PR close?
a + b = a#26623.Rationale for this change
A sort-merge join on
(a, b)fails withSanityCheckPlanwhen a filter below it equates an expression with one of its own sub-expressions, such asx.a + x.b = x.a.The filter puts
a + bandain one equivalence class, witha + bas the canonical expression because it comes first.normalize_exprrewrites bottom-up, so it replaces theainsidea + bwitha + band turnsa + binto(a + b) + b. The join requires[a, b], which normalizes to[a + b, b], while the sort below it is on[a + b, b], which normalizes to[(a + b) + b, b], so the sort no longer satisfies the join. Written asx.a = x.a + x.b, the query works, becauseais the canonical expression.What changes are included in this PR?
When a member of an equivalence class is a sub-expression of the canonical expression, that member now becomes the canonical expression.
EquivalenceClass::new,pushandextendall do it, so every way of building a class gets it.normalize_expris unchanged, so classes without such a pair normalize as before.Two classes that refer to each other (
a + b = c AND c + b = a) still fail; that is #27537.What is the testing strategy for this PR?
The query from the issue is added to
sort_merge_join.slt, with two more that reach the other arms ofadd_equal_conditions.equivalence_properties.sltgets the window case:PARTITION BY a + b ORDER BY aunderWHERE a + b = a. All four fail without the fix, and each has anEXPLAIN.equivalence_properties.sltalso gets a plan thatmainalready gets right and that a fix innormalize_expritself would break: witha = b AND c = a + 1aboveORDER BY a + 1 LIMIT 100, the outerORDER BY a + 1needs no sort.Two unit tests cover classes built with
EquivalenceClass::newor merged byEquivalenceGroup::extend, and a promotion that takes several rounds.Are there any user-facing changes?
No API changes. Queries like the one in the issue now plan instead of failing in
SanityCheckPlan.