Skip to content

fix: keep a sub-expression as the canonical expression of its equivalence class - #26967

Merged
jayzhan211 merged 5 commits into
apache:mainfrom
abp002:smj-self-referencing-equality
Nov 21, 2026
Merged

jayzhan211 merged 5 commits into
apache:mainfrom
abp002:smj-self-referencing-equality

Conversation

@abp002

@abp002 abp002 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A sort-merge join on (a, b) fails with SanityCheckPlan when a filter below it equates an expression with one of its own sub-expressions, such as x.a + x.b = x.a.

The filter puts a + b and a in one equivalence class, with a + b as the canonical expression because it comes first. normalize_expr rewrites bottom-up, so it replaces the a inside a + b with a + b and turns a + b into (a + b) + b. The join requires [a, b], which normalizes to [a + b, b], while the sort below it is on [a + b, b], which normalizes to [(a + b) + b, b], so the sort no longer satisfies the join. Written as x.a = x.a + x.b, the query works, because a is the canonical expression.

What changes are included in this PR?

When a member of an equivalence class is a sub-expression of the canonical expression, that member now becomes the canonical expression. EquivalenceClass::new, push and extend all do it, so every way of building a class gets it. normalize_expr is unchanged, so classes without such a pair normalize as before.

Two classes that refer to each other (a + b = c AND c + b = a) still fail; that is #27537.

What is the testing strategy for this PR?

The query from the issue is added to sort_merge_join.slt, with two more that reach the other arms of add_equal_conditions. equivalence_properties.slt gets the window case: PARTITION BY a + b ORDER BY a under WHERE a + b = a. All four fail without the fix, and each has an EXPLAIN.

equivalence_properties.slt also gets a plan that main already gets right and that a fix in normalize_expr itself would break: with a = b AND c = a + 1 above ORDER BY a + 1 LIMIT 100, the outer ORDER BY a + 1 needs no sort.

Two unit tests cover classes built with EquivalenceClass::new or merged by EquivalenceGroup::extend, and a promotion that takes several rounds.

Are there any user-facing changes?

No API changes. Queries like the one in the issue now plan instead of failing in SanityCheckPlan.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Oct 31, 2026

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 thanks for tracking this down! The fix works, and on current main it also fixes the window case and #27287. I left three small asks inline: run the promotion in EquivalenceClass::push/extend so classes built through EquivalenceGroup::new or extend get it too, and add tests for the window shape and for the merge/push arms of add_equal_conditions.

/// of this class. Otherwise, with `a + b` as the canonical expression of a
/// class that also holds `a`, normalization would replace the `a` inside
/// `a + b` and turn `a + b` into `(a + b) + b`.
fn promote_sub_expressions(&mut self) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 thanks for the fix! Classes built through EquivalenceGroup::new or merged by EquivalenceGroup::extend (via bridge_classes) skip this promotion, so EquivalenceGroup::new([EquivalenceClass::new([a + b, a])]) still normalizes a to (a + b) + b. Both are public. Could you run the promotion inside EquivalenceClass::push and extend instead? Then every way of building a class gets it, and the four calls in add_equal_conditions go away:

         self.exprs.insert(expr);
+        self.promote_sub_expressions();
     }
@@
     pub fn extend(&mut self, other: Self) {
         self.exprs.extend(other.exprs);
+        self.promote_sub_expressions();
@@
                 self.classes[first_idx].extend(other_class);
-                self.classes[first_idx].promote_sub_expressions();
@@
                 self.classes[group_idx].push(right);
-                self.classes[group_idx].promote_sub_expressions();
@@
                 self.classes[group_idx].push(left);
-                self.classes[group_idx].promote_sub_expressions();
@@
-                let mut class = EquivalenceClass::new([left, right]);
-                class.promote_sub_expressions();
+                let class = EquivalenceClass::new([left, right]);

With this change on current main, the physical-expr unit tests and the full sqllogictest suite pass.

Unit test for both paths (fails on the current PR, passes with the change)
    #[test]
    fn test_sub_expression_is_canonical_on_every_path() -> Result<()> {
        let schema = abc_schema();
        let (a, b, c) = (col("a", &schema)?, col("b", &schema)?, col("c", &schema)?);
        let a_plus_b = binary(Arc::clone(&a), Operator::Plus, b, &schema)?;

        // A class built directly, with the larger expression first
        let group = EquivalenceGroup::new([EquivalenceClass::new([
            Arc::clone(&a_plus_b),
            Arc::clone(&a),
        ])]);
        assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a));

        // Two classes bridged by `extend` through the shared `c`
        let mut group = EquivalenceGroup::default();
        group.add_equal_conditions(Arc::clone(&a_plus_b), Arc::clone(&c));
        let mut other = EquivalenceGroup::default();
        other.add_equal_conditions(c, Arc::clone(&a));
        group.extend(other);
        assert!(group.normalize_expr(a_plus_b).eq(&a));
        Ok(())
    }

Comment on lines +247 to +248
statement ok
DROP TABLE eq_in;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 the same bug also breaks a window on main, without a sort-merge join: PARTITION BY a + b ORDER BY a under WHERE a + b = a fails at runtime with Expects PARTITION BY expression to be ordered. This PR fixes it, but nothing tests that path. Could you add it?

Suggested change
statement ok
DROP TABLE eq_in;
statement ok
DROP TABLE eq_in;
# With a + b = a, the window's PARTITION BY a + b is the same key as the
# ORDER BY a below it
statement ok
CREATE TABLE eq_sub (a INT, b INT) AS VALUES (1, 0), (2, 3), (1, 0), (5, 0);
query II
SELECT a, sum(b) OVER (PARTITION BY a + b ORDER BY a) FROM eq_sub WHERE a + b = a ORDER BY a;
----
1 0
1 0
5 0
statement ok
DROP TABLE eq_sub;

1 0 1 0

statement ok
DROP TABLE smj_self_eq;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the promotion after the merge (extend) or after push(right) in add_equal_conditions still passes the tests here, because the issue query only reaches the new-class arm. These two queries reach the other arms. Both fail with SanityCheckPlan on main, and each catches one of those removals:

Suggested change
DROP TABLE smj_self_eq;
DROP TABLE smj_self_eq;
statement ok
CREATE TABLE smj_self_eq_cd (a INT, b INT, c INT, d INT) AS VALUES (1, 0, 1, 1), (2, 3, 5, 2);
# a joins the class of a + b after the class exists
query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
1 0 1 0
# a reaches the class of a + b when two classes merge
query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
1 0 1 0
statement ok
DROP TABLE smj_self_eq_cd;

statement ok
DROP TABLE eq_t;

# The two IN lists hold the same values in a different order, so they compare

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: this test doesn't reach the loop bound in promote_sub_expressions. With the bound replaced by loop, the query still finishes, because neither IN list is inside the other. A promoted expression is always strictly inside the previous canonical expression, so it can't cycle. I'd drop this test, or change its comment so it doesn't claim to cover termination.

@abp002
abp002 force-pushed the smj-self-referencing-equality branch from 7b2d9b2 to a2bc6db Compare November 12, 2026 21:06
@abp002

abp002 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review, and for the tests. I moved the promotion into EquivalenceClass::push and extend, so the four calls in add_equal_conditions are gone, and added the unit test, the window case and the two sort-merge join queries. All four queries fail on main and pass here.

I also dropped the IN test: as you say, it still finishes with the bound replaced by loop, so it wasn't covering termination.

Rebased on main.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 thanks for the updates, all four look good. The new queries fail on main and pass here, and the full sqllogictest suite passes. I left one optional test suggestion inline.

fn promote_sub_expressions(&mut self) {
// Each promoted expression is inside the previous canonical
// expression, so there are at most as many rounds as members.
for _ in 0..self.exprs.len() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: no test needs a second round of this loop. With for _ in 0..1 every test still passes. A second round happens when extend brings in a + b and a behind an unrelated canonical expression. This test passes here and fails with one round:

#[test]
fn test_sub_expression_promotion_takes_several_rounds() -> Result<()> {
    let schema = abc_schema();
    let (a, b, c) = (col("a", &schema)?, col("b", &schema)?, col("c", &schema)?);
    let a_plus_b = binary(Arc::clone(&a), Operator::Plus, Arc::clone(&b), &schema)?;
    let a_plus_b_plus_c = binary(
        Arc::clone(&a_plus_b),
        Operator::Plus,
        Arc::clone(&c),
        &schema,
    )?;

    // `b * c` is canonical in `other`, so `a + b` stays ahead of `a` there
    let b_times_c = binary(b, Operator::Multiply, c, &schema)?;
    let other = EquivalenceClass::new([b_times_c, a_plus_b, Arc::clone(&a)]);
    let mut class = EquivalenceClass::new([a_plus_b_plus_c]);
    class.extend(other);
    assert!(class.canonical_expr().unwrap().eq(&a));
    Ok(())
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, thanks!

cetra3 added a commit to pydantic/datafusion that referenced this pull request Nov 16, 2026
apache#26967 fixes apache#27287 in EquivalenceClass, so the scan no longer needs to
keep the literal out of the class. The scan change also added sorts to
plans that rely on equalities with no column on either side.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this fixes the case in the issue. The same failure still happens when two classes refer to each other:

set datafusion.optimizer.prefer_hash_join = false;
CREATE TABLE t (a BIGINT, b BIGINT, c BIGINT) AS VALUES (1, 0, 1), (2, 3, 5);
SELECT * FROM t x JOIN t y ON x.a = y.a AND x.b = y.b
WHERE x.a + x.b = x.c AND x.c + x.b = x.a;
SanityCheckPlan: ... does not satisfy order requirements: [a@0 ASC, b@1 ASC].
Child-0 order: [[c@2 + b@1 ASC, b@1 ASC]]

With prefer_hash_join = true it returns 1 0 1 1 0 1.

The classes are {a + b, c} and {c + b, a}. Neither canonical expression contains a member of its own class, so promote_sub_expressions does nothing. But each one contains the other class's member, so normalize_expr is still not idempotent:

a  -> c + b -> (a + b) + b -> ...

The general rule is: no canonical expression may contain a member of any class. This PR enforces it inside one class only.

I am fine merging this as a step forward. Could we open a follow-up for the general fix? Some prior art:

System Representative Rewrites members by other members?
Trino EqualityInference cheapest: fewest symbols, then smallest tree no, it unions derived rewrites
PostgreSQL equivclass.c none: pathkeys point at the class no
CockroachDB equiv_set.go lowest column ID (columns only) n/a

A possible shape for the follow-up:

  1. Pick the canonical expression by cost, like Trino: fewest columns, then fewest nodes, then a stable tiebreak. That makes a beat a + b without a special promotion pass.
  2. When a class is added or merged, check the canonical expressions across classes. If a canonical expression contains a member of another class (directly or through a cycle), choose a different canonical expression, or skip that rewrite.
  3. Add debug_assert_eq!(normalize(normalize(e)), normalize(e)) in normalize_expr, so the next case fails in tests instead of in SanityCheckPlan.

Two small things:

  • The PR description mentions a test with two IN lists in a different order. I cannot find it in the diff.
  • Could the slt tests include an EXPLAIN? Today they only show that the query no longer errors.

}
}
self.exprs.insert(expr);
self.promote_sub_expressions();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EquivalenceClass::new calls push per member, so building a class of n members runs n promotion passes. new could insert everything and promote once, as extend does.

Comment on lines +230 to +243
let children = canonical.children();
let Some(idx) = self.exprs.iter().skip(1).position(|expr| {
// Only the children are searched: an expression equal to the
// canonical expression itself is not inside it. The unwrap is
// safe because the closure always returns `Ok`.
children
.iter()
.any(|child| child.exists(|node| Ok(node == expr)).unwrap())
}) else {
return;
};
// `idx` skips the canonical expression, so the expression to
// promote is at `idx + 1`.
self.exprs.move_index(idx + 1, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the real index removes the + 1 and its comment:

Suggested change
let children = canonical.children();
let Some(idx) = self.exprs.iter().skip(1).position(|expr| {
// Only the children are searched: an expression equal to the
// canonical expression itself is not inside it. The unwrap is
// safe because the closure always returns `Ok`.
children
.iter()
.any(|child| child.exists(|node| Ok(node == expr)).unwrap())
}) else {
return;
};
// `idx` skips the canonical expression, so the expression to
// promote is at `idx + 1`.
self.exprs.move_index(idx + 1, 0);
let children = canonical.children();
// Only the children are searched: an expression equal to the
// canonical expression itself is not inside it.
let Some(idx) = self.exprs.iter().enumerate().skip(1).find_map(|(idx, expr)| {
children
.iter()
.any(|child| child.exists(|node| Ok(node == expr)).unwrap())
.then_some(idx)
}) else {
return;
};
self.exprs.move_index(idx, 0);

Arc::clone(&a_plus_b),
Arc::clone(&a),
])]);
assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert_eq! prints both sides on failure:

Suggested change
assert!(group.normalize_expr(Arc::clone(&a_plus_b)).eq(&a));
assert_eq!(&group.normalize_expr(Arc::clone(&a_plus_b)), &a);

(Same for L1790 and L1811.)

statement ok
RESET datafusion.execution.batch_size;

# Regression test for #26623: a filter that equates an expression with one of

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Regression test for #26623: a filter that equates an expression with one of
# Regression test for https://github.com/apache/datafusion/issues/26623: a filter that equates an expression with one of

statement ok
CREATE TABLE eq_t (a BIGINT, b BIGINT, c BIGINT) AS VALUES (1, 1, 2), (3, 3, 4), (5, 5, 0), (2, 2, 3);

# The filter puts a and b in one class, and c and a + 1 in another. The inner

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This query has no sub-expression in a class. It guards against a fix inside normalize_expr. Could the comment say so?

@abp002

abp002 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Pushed:

  • EquivalenceClass::new inserts all the members and promotes once.
  • Applied the suggestions: the real index, assert_eq! and the two slt comments.
  • Each new slt query has an EXPLAIN.
  • The description no longer mentions the IN test, and says where the promotion runs now.

I opened #27537 for the two classes that refer to each other, with your query and the three steps.

@adriangb

Copy link
Copy Markdown
Contributor

Thanks! @jayzhan211 are you happy to merge this?

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 thanks, the round-3 changes look good. One test gap is left, explained inline: the push promotion can be deleted without any test failing. Once that is in, I'm happy to merge (answering @adriangb).

Comment on lines +1171 to +1223
# a joins the class of a + b after the class exists
query TT
EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b
04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a
05)--------TableScan: smj_self_eq_cd projection=[a, b, c]
06)--SubqueryAlias: y
07)----TableScan: smj_self_eq_cd projection=[a, b]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0, projection=[a@0, b@1]
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]

query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
1 0 1 0

# a reaches the class of a + b when two classes merge
query TT
EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b
04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c
05)--------TableScan: smj_self_eq_cd projection=[a, b, c, d]
06)--SubqueryAlias: y
07)----TableScan: smj_self_eq_cd projection=[a, b]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2, projection=[a@0, b@1]
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]

query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
1 0 1 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abp002 thanks for the updates! With new no longer calling push, deleting self.promote_sub_expressions() from push now passes every test. These two queries select only x.a, x.b, so the filter gets projection=[a@0, b@1], and that projection rebuilds the class through EquivalenceClass::new, which promotes anyway. With SELECT * there is no projection, so removing the promotion from push fails the first query, removing it from extend fails the second, and both still fail on main:

Suggested change
# a joins the class of a + b after the class exists
query TT
EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b
04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a
05)--------TableScan: smj_self_eq_cd projection=[a, b, c]
06)--SubqueryAlias: y
07)----TableScan: smj_self_eq_cd projection=[a, b]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0, projection=[a@0, b@1]
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]
query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
1 0 1 0
# a reaches the class of a + b when two classes merge
query TT
EXPLAIN SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Projection: smj_self_eq_cd.a, smj_self_eq_cd.b
04)------Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c
05)--------TableScan: smj_self_eq_cd projection=[a, b, c, d]
06)--SubqueryAlias: y
07)----TableScan: smj_self_eq_cd projection=[a, b]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2, projection=[a@0, b@1]
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]
query IIII
SELECT x.a, x.b, y.a, y.b FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
1 0 1 0
# Both queries keep every column of x: a projection above the filter would
# rebuild the class through EquivalenceClass::new, which promotes on its own
# a joins the class of a + b after the class exists
query TT
EXPLAIN SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.c = smj_self_eq_cd.a
04)------TableScan: smj_self_eq_cd projection=[a, b, c, d]
05)--SubqueryAlias: y
06)----TableScan: smj_self_eq_cd projection=[a, b, c, d]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND c@2 = a@0
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]
query IIIIIIII
SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.c = x.a;
----
1 0 1 1 1 0 1 1
# a reaches the class of a + b when two classes merge
query TT
EXPLAIN SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
logical_plan
01)Inner Join: x.a = y.a, x.b = y.b
02)--SubqueryAlias: x
03)----Filter: smj_self_eq_cd.a + smj_self_eq_cd.b = smj_self_eq_cd.c AND smj_self_eq_cd.d = smj_self_eq_cd.a AND smj_self_eq_cd.d = smj_self_eq_cd.c
04)------TableScan: smj_self_eq_cd projection=[a, b, c, d]
05)--SubqueryAlias: y
06)----TableScan: smj_self_eq_cd projection=[a, b, c, d]
physical_plan
01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0), (b@1, b@1)]
02)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
04)------FilterExec: a@0 + b@1 = c@2 AND d@3 = a@0 AND d@3 = c@2
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
06)--SortExec: expr=[a@0 ASC, b@1 ASC], preserve_partitioning=[true]
07)----RepartitionExec: partitioning=Hash([a@0, b@1], 4), input_partitions=1
08)------DataSourceExec: partitions=1, partition_sizes=[1]
query IIIIIIII
SELECT * FROM smj_self_eq_cd x JOIN smj_self_eq_cd y ON x.a = y.a AND x.b = y.b WHERE x.a + x.b = x.c AND x.a = x.d AND x.d = x.c;
----
1 0 1 1 1 0 1 1

@abp002

abp002 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, @jayzhan211! Applied the suggestion: both queries now use SELECT *. I checked that removing promote_sub_expressions() from push fails the first one and removing it from extend fails the second.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @abp002 @adriangb !

@jayzhan211
jayzhan211 added this pull request to the merge queue Nov 21, 2026
Merged via the queue into apache:main with commit 5476776 Nov 21, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SanityCheckPlan error with sort-merge join when a filter says a + b = a

3 participants