Skip to content

fix: project a dropped column through an equivalent expression - #27319

Open
cetra3 wants to merge 2 commits into
apache:mainfrom
pydantic:fix-projection-pushdown-partitioning
Open

cetra3 wants to merge 2 commits into
apache:mainfrom
pydantic:fix-projection-pushdown-partitioning

Conversation

@cetra3

@cetra3 cetra3 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A query that groups by a struct field, and filters with ... OR field IN (subquery), fails to plan since #26747:

SanityCheckPlan: order and distribution requirements stopped holding after physical optimizer rule 'ProjectionPushdown'

#26747 records get_field(s, k) ≡ __datafusion_extracted_4 when it projects equivalences. EnsureRequirements uses that to reuse the join's hash partitioning for the group key, and the plan is valid. Then ProjectionPushdown gives the join a projection that drops __datafusion_extracted_4 but keeps s. The join cannot project its partitioning through that projection, because the only way to project a dropped column was through a mapping source in its class. So the partitioning becomes an unknown column, and the final check rejects the plan.

What changes are included in this PR?

  • EquivalenceGroup::project_expr and project_expressions now fall back to another member of the expression's equivalence class when the expression itself cannot be projected. In the issue's plan, the join's partitioning on __datafusion_extracted_4 projects to get_field(s, k), which the projection above maps to the group key.
  • The fallback skips literal members, because a literal says nothing about how rows are distributed.

The members of a class are equal on every row, so a member describes the same partitioning or ordering as the expression it replaces. #26747 already removes computed members on a join side that is padded with NULLs, so the fallback cannot pick a member that stops being equal after an outer join.

What is the testing strategy for this PR?

  • A sqllogictest case in equivalence_properties.slt runs the query from the issue. It checks the rows and the plan, which keeps the single-phase aggregate with no extra repartition. It fails on main with the SanityCheckPlan error.
  • A unit test in class.rs, test_project_dropped_column_through_equivalent_expression, checks that a projection that drops a projects it through a = b + c. It fails without the change.
  • No other expected plan in the sqllogictest suite changes.

Are there any user-facing changes?

No. A query that failed to plan now plans.


Claude Code (an AI coding assistant) wrote this change on behalf of cetra3.

A projection that drops a column now projects it through another member of its equivalence class. So a hash join that drops its key column keeps its output partitioning when the key equals an expression the join still outputs.

Closes apache#27318.
cetra3 added a commit to pydantic/datafusion that referenced this pull request Nov 11, 2026
A projection that drops a column now projects it through another member of its equivalence class. So a hash join that drops its key column keeps its output partitioning when the key equals an expression the join still outputs.

Closes apache#27318.

Backport of apache#27319 (open).
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Nov 11, 2026

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cetra3 thanks for tracking this down! One blocking issue inline: a join key computed from the same field (s['k'] || 'z') plans on main but fails to plan with this PR. The other two comments are optional.

aug_mapping: &AugmentedMapping,
expr: &Arc<dyn PhysicalExpr>,
) -> Option<Arc<dyn PhysicalExpr>> {
Self::project_expr_indirect(aug_mapping, expr).or_else(|| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cetra3 thanks for the quick fix! The fallback only runs at the top level, so a join key computed from the field (s['k'] || 'z') still fails, and with this PR that query fails to plan, while main plans it with a two-phase aggregate. With the fallback, EnsureRequirements now reuses the join's partitioning. After ProjectionPushdown, the key __datafusion_extracted_4 || 'z' can only be projected if its child __datafusion_extracted_4 also falls back to get_field(s, 'k').

SELECT s['k'] || 'z' FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || 'z' IN (SELECT s['k'] || 'z' FROM mark_join)
GROUP BY 1;
-- main: az, bz; this PR: SanityCheckPlan error

Moving the fallback into project_expr_indirect covers children too. It needs a guard, because classes can refer to each other (a = b + 1, b = a + 1), and without one that case overflows the stack:

     fn project_expr_indirect(
         aug_mapping: &AugmentedMapping,
         expr: &Arc<dyn PhysicalExpr>,
+        equivalents: Option<&Self>,
+        tried: &mut Vec<Arc<dyn PhysicalExpr>>,
     ) -> Option<Arc<dyn PhysicalExpr>> {
@@
-                        Self::project_expr_indirect(aug_mapping, child)
+                        Self::project_expr_indirect(
+                            aug_mapping,
+                            child,
+                            equivalents,
+                            tried,
+                        )
@@
-        None
+        // Members of a class are equal on every row, so a projectable member
+        // stands in for `expr`. `tried` stops cycles such as `x = f(y)`, `y = g(x)`.
+        let group = equivalents?;
+        if tried.contains(expr) {
+            return None;
+        }
+        tried.push(Arc::clone(expr));
+        group
+            .get_equivalence_class(expr)?
+            .iter()
+            .filter(|member| *member != expr && !member.is::<Literal>())
+            .find_map(|member| {
+                Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
+            })
     }
@@ fn project_expr_or_equivalent
-        Self::project_expr_indirect(aug_mapping, expr).or_else(|| {
-            self.get_equivalence_class(expr)?
-                .iter()
-                .filter(|member| *member != expr && !member.is::<Literal>())
-                .find_map(|member| Self::project_expr_indirect(aug_mapping, member))
-        })
+        Self::project_expr_indirect(aug_mapping, expr, Some(self), &mut Vec::new())
@@ fn project
-                if let Some(projected) = Self::project_expr_indirect(augmented, source) {
+                if let Some(projected) =
+                    Self::project_expr_indirect(augmented, source, None, &mut Vec::new())
+                {

With this change and the tests below, the full sqllogictest suite and cargo clippy -p datafusion-physical-expr --all-targets --all-features pass. Each test fails if you remove the part of the change it covers.

Tests to add: the slt case above, a nested unit test, and a cycle unit test

In equivalence_properties.slt, before DROP TABLE mark_join;:

# The same plan with a join key computed from the field.
query T rowsort
SELECT s['k'] || 'z' FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || 'z' IN (SELECT s['k'] || 'z' FROM mark_join)
GROUP BY 1;
----
az
bz

In class.rs tests:

    #[test]
    fn test_project_dropped_child_through_equivalent_expression() -> Result<()> {
        // - columns: [a, b, c, d], with a = b + c and d = a * 2.
        // - a projection that keeps b and c projects d to (b + c) * 2, which
        //   needs a's equivalent inside d's equivalent.
        let schema = abc_schema();
        let a = col("a", &schema)?;
        let b_plus_c = binary(
            col("b", &schema)?,
            Operator::Plus,
            col("c", &schema)?,
            &schema,
        )?;
        let mut group = EquivalenceGroup::default();
        group.add_equal_conditions(Arc::clone(&a), b_plus_c);
        group.add_equal_conditions(
            col("d", &schema)?,
            binary(a, Operator::Multiply, lit(2), &schema)?,
        );
        let mapping = ProjectionMapping::from_indices(&[1, 2], &schema)?;
        let projected_schema = Arc::new(Schema::new(vec![
            Field::new("b", DataType::Int32, false),
            Field::new("c", DataType::Int32, false),
        ]));
        let expected = binary(
            binary(
                col("b", &projected_schema)?,
                Operator::Plus,
                col("c", &projected_schema)?,
                &projected_schema,
            )?,
            Operator::Multiply,
            lit(2),
            &projected_schema,
        )?;

        let projected = group.project_expr(&mapping, &col("d", &schema)?);
        assert!(projected.is_some_and(|expr| expr.eq(&expected)));
        Ok(())
    }

    #[test]
    fn test_project_cyclic_equivalent_expressions_terminates() -> Result<()> {
        // a = b + 1 and b = a + 1 refer to each other; a projection that keeps
        // only c cannot project a, and must not recurse forever.
        let schema = abc_schema();
        let (a, b) = (col("a", &schema)?, col("b", &schema)?);
        let mut group = EquivalenceGroup::default();
        group.add_equal_conditions(
            Arc::clone(&a),
            binary(Arc::clone(&b), Operator::Plus, lit(1), &schema)?,
        );
        group.add_equal_conditions(
            Arc::clone(&b),
            binary(Arc::clone(&a), Operator::Plus, lit(1), &schema)?,
        );
        let mapping = ProjectionMapping::from_indices(&[2], &schema)?;
        assert!(group.project_expr(&mapping, &a).is_none());
        Ok(())
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: s['k'] || 'z' plans without this PR and failed with it. I took your patch in 49e86ad with one change.

With tried as a list, a key that uses the dropped column twice still failed, because the second use was treated as a cycle:

SELECT s['k'] || s['k'] FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || s['k'] IN (SELECT s['k'] || s['k'] FROM mark_join)
GROUP BY 1;
-- SanityCheckPlan error

tried is now a map from each expression to its result. An entry of None while the expression is in progress stops cycles, and the stored result serves later uses of the same expression. I added that query to the slt file and a unit test for it (test_project_dropped_child_used_twice), along with your slt case and your two unit tests.

08)----------RepartitionExec: partitioning=Hash([mark_join.s[k]@0], 2), input_partitions=2
09)------------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/2.parquet]]}, projection=[get_field(s@0, k) as mark_join.s[k]], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible

statement ok

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: a UNION ALL of two of these joins fails differently on main (Internal error: ... Not all InterleaveExec children have a consistent hash or range partitioning). This PR fixes it, but no test covers that shape. Could you add it here?

Suggested change
statement ok
# A union of two such joins, which InterleaveExec combines by partitioning.
query T rowsort
SELECT k FROM (
SELECT s['k'] AS k FROM mark_join
WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM mark_join)
UNION ALL
SELECT s['k'] FROM mark_join
WHERE s['k'] = 'b' OR s['k'] IN (SELECT s['k'] FROM mark_join)
) GROUP BY k;
----
a
b
statement ok

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 49e86ad. I confirmed it fails with the InterleaveExec error without the fix and returns a, b with it.

Self::project_expr_indirect(aug_mapping, expr).or_else(|| {
self.get_equivalence_class(expr)?
.iter()
.filter(|member| *member != expr && !member.is::<Literal>())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: removing && !member.is::<Literal>() still passes every test in this PR and the equivalence, join, aggregate and subquery slt files, so nothing pins this skip. Also, if x is in a class with 'a', every row has the same x, so Hash(['a']) would still be true. Could you either drop the skip and its doc sentence, or add a test that shows what it prevents?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I dropped the skip and its doc sentence in 49e86ad. The tests still pass without it.

The fallback to another member of an equivalence class now also applies to
the children of an expression, so a join key computed from a dropped column
keeps its partitioning. A map of results stops cycles between classes and
serves an expression that occurs more than once.

The fallback no longer skips literal members.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cetra3 thanks for taking the earlier suggestions. One blocking comment inline: the tried map from my round-1 patch makes planning cubic in the size of an equivalence class, with a small fix.

Optional: the PR description still says "The fallback skips literal members", which 49e86ad removed, and the testing section lists only the first slt case and the first unit test. Could you drop that bullet and add the compound key (s['k'] || 'z'), repeated column (s['k'] || s['k']) and UNION ALL slt cases plus the three new unit tests?

Comment on lines +690 to +706
let group = equivalents?;
if let Some(known) = tried.get(expr) {
return known.clone();
}
// Until `expr` has a result, `None` stops cycles such as `x = f(y)`,
// `y = g(x)`. The stored result then serves later uses of `expr`, such
// as the second `x` in `x * x`.
tried.insert(Arc::clone(expr), None);
let projected = group.get_equivalence_class(expr).and_then(|cls| {
cls.iter()
.filter(|member| *member != expr)
.find_map(|member| {
Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
})
});
tried.insert(Arc::clone(expr), projected.clone());
projected

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cetra3 thanks for the update, and sorry: the tried shape from my patch makes project() cubic in class size, because each dropped member of a class searches the whole class again. EXPLAIN on SELECT c0 FROM w WHERE c1 = c2 AND c2 = c3 AND ... goes from 26 to 99 ms with 64 equated columns and from 101 to 684 ms with 128 (debug build). Keying tried by class index brings it to 30 / 114 ms, and the PR's tests, the full slt suite and clippy pass with it:

@@ fn project_expr_indirect
     /// If `equivalents` is given and `expr` still cannot be projected, this
     /// projects another member of the class of `expr` in that group. `tried`
-    /// holds the result of that step for each expression, for one top-level
-    /// call.
+    /// holds the result of that step for each class, keyed by its index in
+    /// `equivalents`, for one top-level call.
     fn project_expr_indirect(
         aug_mapping: &AugmentedMapping,
         expr: &Arc<dyn PhysicalExpr>,
         equivalents: Option<&Self>,
-        tried: &mut HashMap<Arc<dyn PhysicalExpr>, Option<Arc<dyn PhysicalExpr>>>,
+        tried: &mut HashMap<usize, Option<Arc<dyn PhysicalExpr>>>,
     ) -> Option<Arc<dyn PhysicalExpr>> {
@@
         let group = equivalents?;
-        if let Some(known) = tried.get(expr) {
+        let &idx = group.map.get(expr)?;
+        if let Some(known) = tried.get(&idx) {
             return known.clone();
         }
-        // Until `expr` has a result, `None` stops cycles such as `x = f(y)`,
-        // `y = g(x)`. The stored result then serves later uses of `expr`, such
-        // as the second `x` in `x * x`.
-        tried.insert(Arc::clone(expr), None);
-        let projected = group.get_equivalence_class(expr).and_then(|cls| {
-            cls.iter()
-                .filter(|member| *member != expr)
-                .find_map(|member| {
-                    Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
-                })
-        });
-        tried.insert(Arc::clone(expr), projected.clone());
+        // Until the class has a result, `None` stops cycles such as `x = f(y)`,
+        // `y = g(x)`. The stored result then serves every later member of the
+        // class, such as the second `x` in `x * x`.
+        tried.insert(idx, None);
+        let projected = group.classes[idx]
+            .iter()
+            .filter(|member| *member != expr)
+            .find_map(|member| {
+                Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
+            });
+        tried.insert(idx, projected.clone());
         projected
Timings, and why the map stays per top-level call

EXPLAIN of SELECT c0 FROM w WHERE c1 = c2 AND ... AND c{n-1} = c{n}, debug datafusion-cli, best of 3:

n main this PR class-keyed tried
8 2 ms 3 ms 3 ms
16 5 ms 6 ms 5 ms
32 9 ms 19 ms 11 ms
64 26 ms 99 ms 30 ms
128 101 ms 684 ms 114 ms

Sharing one tried across a whole project_expressions call would be linear, but a None stored while another class was still in progress then sticks. With a = b + 1 = d * 2, b = a - 1 and a projection that keeps only d, project_expressions([a, b]) gives b = None, while this PR and the per-call version give d * 2 - 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SanityCheckPlan fails after ProjectionPushdown for GROUP BY over a mark join on a struct field (regression from #26747)

2 participants