Repository navigation
Conversation
A projection that drops a column now projects it through another member of its equivalence class. So a hash join that drops its key column keeps its output partitioning when the key equals an expression the join still outputs. Closes apache#27318.
A projection that drops a column now projects it through another member of its equivalence class. So a hash join that drops its key column keeps its output partitioning when the key equals an expression the join still outputs. Closes apache#27318. Backport of apache#27319 (open).
jayzhan211
left a comment
There was a problem hiding this comment.
@cetra3 thanks for tracking this down! One blocking issue inline: a join key computed from the same field (s['k'] || 'z') plans on main but fails to plan with this PR. The other two comments are optional.
| aug_mapping: &AugmentedMapping, | ||
| expr: &Arc<dyn PhysicalExpr>, | ||
| ) -> Option<Arc<dyn PhysicalExpr>> { | ||
| Self::project_expr_indirect(aug_mapping, expr).or_else(|| { |
There was a problem hiding this comment.
@cetra3 thanks for the quick fix! The fallback only runs at the top level, so a join key computed from the field (s['k'] || 'z') still fails, and with this PR that query fails to plan, while main plans it with a two-phase aggregate. With the fallback, EnsureRequirements now reuses the join's partitioning. After ProjectionPushdown, the key __datafusion_extracted_4 || 'z' can only be projected if its child __datafusion_extracted_4 also falls back to get_field(s, 'k').
SELECT s['k'] || 'z' FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || 'z' IN (SELECT s['k'] || 'z' FROM mark_join)
GROUP BY 1;
-- main: az, bz; this PR: SanityCheckPlan errorMoving the fallback into project_expr_indirect covers children too. It needs a guard, because classes can refer to each other (a = b + 1, b = a + 1), and without one that case overflows the stack:
fn project_expr_indirect(
aug_mapping: &AugmentedMapping,
expr: &Arc<dyn PhysicalExpr>,
+ equivalents: Option<&Self>,
+ tried: &mut Vec<Arc<dyn PhysicalExpr>>,
) -> Option<Arc<dyn PhysicalExpr>> {
@@
- Self::project_expr_indirect(aug_mapping, child)
+ Self::project_expr_indirect(
+ aug_mapping,
+ child,
+ equivalents,
+ tried,
+ )
@@
- None
+ // Members of a class are equal on every row, so a projectable member
+ // stands in for `expr`. `tried` stops cycles such as `x = f(y)`, `y = g(x)`.
+ let group = equivalents?;
+ if tried.contains(expr) {
+ return None;
+ }
+ tried.push(Arc::clone(expr));
+ group
+ .get_equivalence_class(expr)?
+ .iter()
+ .filter(|member| *member != expr && !member.is::<Literal>())
+ .find_map(|member| {
+ Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
+ })
}
@@ fn project_expr_or_equivalent
- Self::project_expr_indirect(aug_mapping, expr).or_else(|| {
- self.get_equivalence_class(expr)?
- .iter()
- .filter(|member| *member != expr && !member.is::<Literal>())
- .find_map(|member| Self::project_expr_indirect(aug_mapping, member))
- })
+ Self::project_expr_indirect(aug_mapping, expr, Some(self), &mut Vec::new())
@@ fn project
- if let Some(projected) = Self::project_expr_indirect(augmented, source) {
+ if let Some(projected) =
+ Self::project_expr_indirect(augmented, source, None, &mut Vec::new())
+ {With this change and the tests below, the full sqllogictest suite and cargo clippy -p datafusion-physical-expr --all-targets --all-features pass. Each test fails if you remove the part of the change it covers.
Tests to add: the slt case above, a nested unit test, and a cycle unit test
In equivalence_properties.slt, before DROP TABLE mark_join;:
# The same plan with a join key computed from the field.
query T rowsort
SELECT s['k'] || 'z' FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || 'z' IN (SELECT s['k'] || 'z' FROM mark_join)
GROUP BY 1;
----
az
bzIn class.rs tests:
#[test]
fn test_project_dropped_child_through_equivalent_expression() -> Result<()> {
// - columns: [a, b, c, d], with a = b + c and d = a * 2.
// - a projection that keeps b and c projects d to (b + c) * 2, which
// needs a's equivalent inside d's equivalent.
let schema = abc_schema();
let a = col("a", &schema)?;
let b_plus_c = binary(
col("b", &schema)?,
Operator::Plus,
col("c", &schema)?,
&schema,
)?;
let mut group = EquivalenceGroup::default();
group.add_equal_conditions(Arc::clone(&a), b_plus_c);
group.add_equal_conditions(
col("d", &schema)?,
binary(a, Operator::Multiply, lit(2), &schema)?,
);
let mapping = ProjectionMapping::from_indices(&[1, 2], &schema)?;
let projected_schema = Arc::new(Schema::new(vec![
Field::new("b", DataType::Int32, false),
Field::new("c", DataType::Int32, false),
]));
let expected = binary(
binary(
col("b", &projected_schema)?,
Operator::Plus,
col("c", &projected_schema)?,
&projected_schema,
)?,
Operator::Multiply,
lit(2),
&projected_schema,
)?;
let projected = group.project_expr(&mapping, &col("d", &schema)?);
assert!(projected.is_some_and(|expr| expr.eq(&expected)));
Ok(())
}
#[test]
fn test_project_cyclic_equivalent_expressions_terminates() -> Result<()> {
// a = b + 1 and b = a + 1 refer to each other; a projection that keeps
// only c cannot project a, and must not recurse forever.
let schema = abc_schema();
let (a, b) = (col("a", &schema)?, col("b", &schema)?);
let mut group = EquivalenceGroup::default();
group.add_equal_conditions(
Arc::clone(&a),
binary(Arc::clone(&b), Operator::Plus, lit(1), &schema)?,
);
group.add_equal_conditions(
Arc::clone(&b),
binary(Arc::clone(&a), Operator::Plus, lit(1), &schema)?,
);
let mapping = ProjectionMapping::from_indices(&[2], &schema)?;
assert!(group.project_expr(&mapping, &a).is_none());
Ok(())
}There was a problem hiding this comment.
Thanks, confirmed: s['k'] || 'z' plans without this PR and failed with it. I took your patch in 49e86ad with one change.
With tried as a list, a key that uses the dropped column twice still failed, because the second use was treated as a cycle:
SELECT s['k'] || s['k'] FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || s['k'] IN (SELECT s['k'] || s['k'] FROM mark_join)
GROUP BY 1;
-- SanityCheckPlan errortried is now a map from each expression to its result. An entry of None while the expression is in progress stops cycles, and the stored result serves later uses of the same expression. I added that query to the slt file and a unit test for it (test_project_dropped_child_used_twice), along with your slt case and your two unit tests.
| 08)----------RepartitionExec: partitioning=Hash([mark_join.s[k]@0], 2), input_partitions=2 | ||
| 09)------------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/2.parquet]]}, projection=[get_field(s@0, k) as mark_join.s[k]], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible | ||
|
|
||
| statement ok |
There was a problem hiding this comment.
Optional: a UNION ALL of two of these joins fails differently on main (Internal error: ... Not all InterleaveExec children have a consistent hash or range partitioning). This PR fixes it, but no test covers that shape. Could you add it here?
| statement ok | |
| # A union of two such joins, which InterleaveExec combines by partitioning. | |
| query T rowsort | |
| SELECT k FROM ( | |
| SELECT s['k'] AS k FROM mark_join | |
| WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM mark_join) | |
| UNION ALL | |
| SELECT s['k'] FROM mark_join | |
| WHERE s['k'] = 'b' OR s['k'] IN (SELECT s['k'] FROM mark_join) | |
| ) GROUP BY k; | |
| ---- | |
| a | |
| b | |
| statement ok |
There was a problem hiding this comment.
Added in 49e86ad. I confirmed it fails with the InterleaveExec error without the fix and returns a, b with it.
| Self::project_expr_indirect(aug_mapping, expr).or_else(|| { | ||
| self.get_equivalence_class(expr)? | ||
| .iter() | ||
| .filter(|member| *member != expr && !member.is::<Literal>()) |
There was a problem hiding this comment.
Optional cleanup: removing && !member.is::<Literal>() still passes every test in this PR and the equivalence, join, aggregate and subquery slt files, so nothing pins this skip. Also, if x is in a class with 'a', every row has the same x, so Hash(['a']) would still be true. Could you either drop the skip and its doc sentence, or add a test that shows what it prevents?
There was a problem hiding this comment.
Agreed, I dropped the skip and its doc sentence in 49e86ad. The tests still pass without it.
The fallback to another member of an equivalence class now also applies to the children of an expression, so a join key computed from a dropped column keeps its partitioning. A map of results stops cycles between classes and serves an expression that occurs more than once. The fallback no longer skips literal members. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
jayzhan211
left a comment
There was a problem hiding this comment.
@cetra3 thanks for taking the earlier suggestions. One blocking comment inline: the tried map from my round-1 patch makes planning cubic in the size of an equivalence class, with a small fix.
Optional: the PR description still says "The fallback skips literal members", which 49e86ad removed, and the testing section lists only the first slt case and the first unit test. Could you drop that bullet and add the compound key (s['k'] || 'z'), repeated column (s['k'] || s['k']) and UNION ALL slt cases plus the three new unit tests?
| let group = equivalents?; | ||
| if let Some(known) = tried.get(expr) { | ||
| return known.clone(); | ||
| } | ||
| // Until `expr` has a result, `None` stops cycles such as `x = f(y)`, | ||
| // `y = g(x)`. The stored result then serves later uses of `expr`, such | ||
| // as the second `x` in `x * x`. | ||
| tried.insert(Arc::clone(expr), None); | ||
| let projected = group.get_equivalence_class(expr).and_then(|cls| { | ||
| cls.iter() | ||
| .filter(|member| *member != expr) | ||
| .find_map(|member| { | ||
| Self::project_expr_indirect(aug_mapping, member, equivalents, tried) | ||
| }) | ||
| }); | ||
| tried.insert(Arc::clone(expr), projected.clone()); | ||
| projected |
There was a problem hiding this comment.
@cetra3 thanks for the update, and sorry: the tried shape from my patch makes project() cubic in class size, because each dropped member of a class searches the whole class again. EXPLAIN on SELECT c0 FROM w WHERE c1 = c2 AND c2 = c3 AND ... goes from 26 to 99 ms with 64 equated columns and from 101 to 684 ms with 128 (debug build). Keying tried by class index brings it to 30 / 114 ms, and the PR's tests, the full slt suite and clippy pass with it:
@@ fn project_expr_indirect
/// If `equivalents` is given and `expr` still cannot be projected, this
/// projects another member of the class of `expr` in that group. `tried`
- /// holds the result of that step for each expression, for one top-level
- /// call.
+ /// holds the result of that step for each class, keyed by its index in
+ /// `equivalents`, for one top-level call.
fn project_expr_indirect(
aug_mapping: &AugmentedMapping,
expr: &Arc<dyn PhysicalExpr>,
equivalents: Option<&Self>,
- tried: &mut HashMap<Arc<dyn PhysicalExpr>, Option<Arc<dyn PhysicalExpr>>>,
+ tried: &mut HashMap<usize, Option<Arc<dyn PhysicalExpr>>>,
) -> Option<Arc<dyn PhysicalExpr>> {
@@
let group = equivalents?;
- if let Some(known) = tried.get(expr) {
+ let &idx = group.map.get(expr)?;
+ if let Some(known) = tried.get(&idx) {
return known.clone();
}
- // Until `expr` has a result, `None` stops cycles such as `x = f(y)`,
- // `y = g(x)`. The stored result then serves later uses of `expr`, such
- // as the second `x` in `x * x`.
- tried.insert(Arc::clone(expr), None);
- let projected = group.get_equivalence_class(expr).and_then(|cls| {
- cls.iter()
- .filter(|member| *member != expr)
- .find_map(|member| {
- Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
- })
- });
- tried.insert(Arc::clone(expr), projected.clone());
+ // Until the class has a result, `None` stops cycles such as `x = f(y)`,
+ // `y = g(x)`. The stored result then serves every later member of the
+ // class, such as the second `x` in `x * x`.
+ tried.insert(idx, None);
+ let projected = group.classes[idx]
+ .iter()
+ .filter(|member| *member != expr)
+ .find_map(|member| {
+ Self::project_expr_indirect(aug_mapping, member, equivalents, tried)
+ });
+ tried.insert(idx, projected.clone());
projectedTimings, and why the map stays per top-level call
EXPLAIN of SELECT c0 FROM w WHERE c1 = c2 AND ... AND c{n-1} = c{n}, debug datafusion-cli, best of 3:
| n | main | this PR | class-keyed tried |
|---|---|---|---|
| 8 | 2 ms | 3 ms | 3 ms |
| 16 | 5 ms | 6 ms | 5 ms |
| 32 | 9 ms | 19 ms | 11 ms |
| 64 | 26 ms | 99 ms | 30 ms |
| 128 | 101 ms | 684 ms | 114 ms |
Sharing one tried across a whole project_expressions call would be linear, but a None stored while another class was still in progress then sticks. With a = b + 1 = d * 2, b = a - 1 and a projection that keeps only d, project_expressions([a, b]) gives b = None, while this PR and the per-call version give d * 2 - 1.
Which issue does this PR close?
Rationale for this change
A query that groups by a struct field, and filters with
... OR field IN (subquery), fails to plan since #26747:#26747 records
get_field(s, k) ≡ __datafusion_extracted_4when it projects equivalences.EnsureRequirementsuses that to reuse the join's hash partitioning for the group key, and the plan is valid. ThenProjectionPushdowngives the join a projection that drops__datafusion_extracted_4but keepss. The join cannot project its partitioning through that projection, because the only way to project a dropped column was through a mapping source in its class. So the partitioning becomes an unknown column, and the final check rejects the plan.What changes are included in this PR?
EquivalenceGroup::project_exprandproject_expressionsnow fall back to another member of the expression's equivalence class when the expression itself cannot be projected. In the issue's plan, the join's partitioning on__datafusion_extracted_4projects toget_field(s, k), which the projection above maps to the group key.The members of a class are equal on every row, so a member describes the same partitioning or ordering as the expression it replaces. #26747 already removes computed members on a join side that is padded with NULLs, so the fallback cannot pick a member that stops being equal after an outer join.
What is the testing strategy for this PR?
sqllogictestcase inequivalence_properties.sltruns the query from the issue. It checks the rows and the plan, which keeps the single-phase aggregate with no extra repartition. It fails onmainwith theSanityCheckPlanerror.class.rs,test_project_dropped_column_through_equivalent_expression, checks that a projection that dropsaprojects it througha = b + c. It fails without the change.sqllogictestsuite changes.Are there any user-facing changes?
No. A query that failed to plan now plans.
Claude Code (an AI coding assistant) wrote this change on behalf of cetra3.