Skip to content

test: cover CASE group key pinned by HAVING with parquet filter pushdown - #27288

Open
cetra3 wants to merge 2 commits into
apache:mainfrom
pydantic:fix-scan-filter-literal-equivalence
Open

cetra3 wants to merge 2 commits into
apache:mainfrom
pydantic:fix-scan-filter-literal-equivalence

Conversation

@cetra3

@cetra3 cetra3 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Posted by Claude (an AI assistant) on behalf of cetra3.

Which issue does this PR close?

Depends on #26967, which contains the fix. This PR only adds a regression test and should land after it.

Rationale for this change

With datafusion.execution.parquet.pushdown_filters = true, a query that groups by a CASE key and pins it with HAVING fails planning with a SanityCheckPlan error. See #27287 for a datafusion-cli reproduction.

The scan turns the pushed-down complex_expr = literal filter into an equivalence class. normalize_expr rewrites bottom-up, so the literal inside complex_expr itself is replaced by the canonical expression of the class, and the aggregate's ordering requirement on the group key no longer matches.

#26967 fixes this in EquivalenceClass: a member that appears inside the canonical expression becomes the canonical expression. An earlier version of this PR instead kept the literal out of the class at the scan. That is no longer needed, and it added sorts to plans that rely on equalities with no column on either side, so it is removed.

What changes are included in this PR?

A regression test in push_down_filter_regression.slt with the query from #27287 and a result assertion.

What is the testing strategy for this PR?

The new test fails on main with the SanityCheckPlan error and passes with #26967 applied.

Are there any user-facing changes?

No.

🤖 Generated with Claude Code

…= literal filters

`FileScanConfig::add_filter_equivalence_info` turned every equality of the
source's exact filter into an equivalence class. For `complex_expr = literal`
that put the literal in a class, and normalization then replaced the literal
inside `complex_expr` with the class's canonical expression. A partial
aggregate that relied on the constant group key then failed `SanityCheckPlan`.

Follow `FilterExec`: only an equality with a column on one side becomes an
equivalence class, and any other equality is recorded as a constant.

Closes apache#27287.

Claude-Session: https://claude.ai/code/session_01USkuh48jV15LVKgyqXmCod
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Nov 10, 2026
cetra3 added a commit to pydantic/datafusion that referenced this pull request Nov 10, 2026
if binary.op() != &Operator::Eq {
continue;
}
// Follow `FilterExec`: only an equality with a column on one side

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we put this sort of logic in a shared place so that we can avoid duplicating diverging bits of code?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @adriangb. I think the simplest way to avoid two diverging copies is to drop this one: #26967 (approved) fixes the root cause in EquivalenceGroup, and with only #26967 on this PR's base, the #27287 query returns y 1 and your new slt test passes unchanged.

Keeping it also costs a sort: with pushdown_filters = true, equalities with no column on either side no longer form a class at the scan, so WHERE a = 'x' AND upper(a) = upper(b) ORDER BY upper(b) gains a SortExec that main doesn't need. Could you drop the mod.rs change, keep the slt test, and land it after #26967?

Plans that gain a sort with this PR (all have none, or fewer keys, on main and on main + #26967)
SET datafusion.execution.parquet.pushdown_filters = true;
-- t: parquet with VARCHAR columns a, b
EXPLAIN SELECT a, b FROM t WHERE a = 'x' AND upper(a) = upper(b) ORDER BY upper(b);         -- SortExec added
EXPLAIN SELECT a, b FROM t WHERE upper(a) = 'X' AND upper(a) = upper(b) ORDER BY upper(b);  -- SortExec + SPM added
EXPLAIN SELECT a, b FROM t WHERE upper(a) = upper(b) ORDER BY upper(a), upper(b);           -- sorts on 2 keys instead of 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I've left the regression test in here, but otherwise let's defer to your fix in 26967

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cetra3 thanks for the fix and the clear write-up. Requesting changes for one reason, explained in the reply on @adriangb's thread: #26967 fixes #27287 at the root, so this PR could keep only the slt test.

Arc::clone(binary.right()),
)?;
} else {
constants.extend(ConstExpr::collect_predicate_constants(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this scan change stays (for example if #26967 stalls), nothing tests this branch: with it removed, every test in this PR still passes. This plan has no SortExec only because of the branch, so could you add it as an EXPLAIN test?

EXPLAIN SELECT a, b FROM t WHERE upper(a) = 'X' ORDER BY upper(a);

apache#26967 fixes apache#27287 in EquivalenceClass, so the scan no longer needs to
keep the literal out of the class. The scan change also added sorts to
plans that rely on equalities with no column on either side.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@cetra3 cetra3 changed the title fix: keep a literal out of scan equivalence classes for complex_expr = literal filters test: cover CASE group key pinned by HAVING with parquet filter pushdown Nov 16, 2026
@github-actions github-actions Bot removed the datasource Changes to the datasource crate label Nov 16, 2026
@cetra3

cetra3 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Note: this is expecting to fail until #26967 is merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REGRESSION: SanityCheckPlan error for GROUP BY ... HAVING on a CASE key with parquet filter pushdown

3 participants