Repository navigation
Conversation
…= literal filters `FileScanConfig::add_filter_equivalence_info` turned every equality of the source's exact filter into an equivalence class. For `complex_expr = literal` that put the literal in a class, and normalization then replaced the literal inside `complex_expr` with the class's canonical expression. A partial aggregate that relied on the constant group key then failed `SanityCheckPlan`. Follow `FilterExec`: only an equality with a column on one side becomes an equivalence class, and any other equality is recorded as a constant. Closes apache#27287. Claude-Session: https://claude.ai/code/session_01USkuh48jV15LVKgyqXmCod
…= literal filters Backport of apache#27288 (open), from its head 8c7f9b3. Claude-Session: https://claude.ai/code/session_01USkuh48jV15LVKgyqXmCod
| if binary.op() != &Operator::Eq { | ||
| continue; | ||
| } | ||
| // Follow `FilterExec`: only an equality with a column on one side |
There was a problem hiding this comment.
Could we put this sort of logic in a shared place so that we can avoid duplicating diverging bits of code?
There was a problem hiding this comment.
Agree with @adriangb. I think the simplest way to avoid two diverging copies is to drop this one: #26967 (approved) fixes the root cause in EquivalenceGroup, and with only #26967 on this PR's base, the #27287 query returns y 1 and your new slt test passes unchanged.
Keeping it also costs a sort: with pushdown_filters = true, equalities with no column on either side no longer form a class at the scan, so WHERE a = 'x' AND upper(a) = upper(b) ORDER BY upper(b) gains a SortExec that main doesn't need. Could you drop the mod.rs change, keep the slt test, and land it after #26967?
Plans that gain a sort with this PR (all have none, or fewer keys, on main and on main + #26967)
SET datafusion.execution.parquet.pushdown_filters = true;
-- t: parquet with VARCHAR columns a, b
EXPLAIN SELECT a, b FROM t WHERE a = 'x' AND upper(a) = upper(b) ORDER BY upper(b); -- SortExec added
EXPLAIN SELECT a, b FROM t WHERE upper(a) = 'X' AND upper(a) = upper(b) ORDER BY upper(b); -- SortExec + SPM added
EXPLAIN SELECT a, b FROM t WHERE upper(a) = upper(b) ORDER BY upper(a), upper(b); -- sorts on 2 keys instead of 1There was a problem hiding this comment.
Done, I've left the regression test in here, but otherwise let's defer to your fix in 26967
| Arc::clone(binary.right()), | ||
| )?; | ||
| } else { | ||
| constants.extend(ConstExpr::collect_predicate_constants( |
There was a problem hiding this comment.
If this scan change stays (for example if #26967 stalls), nothing tests this branch: with it removed, every test in this PR still passes. This plan has no SortExec only because of the branch, so could you add it as an EXPLAIN test?
EXPLAIN SELECT a, b FROM t WHERE upper(a) = 'X' ORDER BY upper(a);apache#26967 fixes apache#27287 in EquivalenceClass, so the scan no longer needs to keep the literal out of the class. The scan change also added sorts to plans that rely on equalities with no column on either side. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Note: this is expecting to fail until #26967 is merged |
Posted by Claude (an AI assistant) on behalf of cetra3.
Which issue does this PR close?
Depends on #26967, which contains the fix. This PR only adds a regression test and should land after it.
Rationale for this change
With
datafusion.execution.parquet.pushdown_filters = true, a query that groups by aCASEkey and pins it withHAVINGfails planning with aSanityCheckPlanerror. See #27287 for adatafusion-clireproduction.The scan turns the pushed-down
complex_expr = literalfilter into an equivalence class.normalize_exprrewrites bottom-up, so the literal insidecomplex_expritself is replaced by the canonical expression of the class, and the aggregate's ordering requirement on the group key no longer matches.#26967 fixes this in
EquivalenceClass: a member that appears inside the canonical expression becomes the canonical expression. An earlier version of this PR instead kept the literal out of the class at the scan. That is no longer needed, and it added sorts to plans that rely on equalities with no column on either side, so it is removed.What changes are included in this PR?
A regression test in
push_down_filter_regression.sltwith the query from #27287 and a result assertion.What is the testing strategy for this PR?
The new test fails on
mainwith theSanityCheckPlanerror and passes with #26967 applied.Are there any user-facing changes?
No.
🤖 Generated with Claude Code