Repository navigation
Revert #25790 (Decimal filter selectivity) due to TPC-H Q18 regression - #27145
Merged
Merged
Conversation
…sis (apache#25790)" This reverts commit 6145a09.
Contributor
|
Ope, appears great minds think alike. I've opened a PR for this as well 😆 I'll go ahead and close mine. |
alamb
approved these changes
Nov 5, 2026
Contributor
There was a problem hiding this comment.
Thank you -- I verified this is a clean revert.
Thank you @asolimando and @devanbenz
alamb
enabled auto-merge
November 5, 2026 21:00
This was referenced Nov 6, 2026
bvolpato
pushed a commit
to bvolpato/datafusion
that referenced
this pull request
Nov 21, 2026
…vity for columns without known bounds (apache#27148) ## Which issue does this PR close? - Closes apache#27146. - Re-lands apache#25790 (reverted in apache#27145 because of apache#27108). ## Rationale for this change This PR does two things: 1. Re-lands apache#25790, which estimates decimal filters with interval analysis. 2. For any type, not just decimals, uses the default selectivity when interval analysis can't estimate a filter, e.g. on a column without min/max such as a `count(*)` or `rank()` output. These filters were estimated at 100% of rows before. apache#25790 let interval analysis estimate decimal filters. When interval analysis can't tell how many values of a column pass, it estimated 100% of rows, which is worse than the 20% default. Q18's `HAVING sum(l_quantity) > 300` hit this: the aggregate output has no min/max, and the bad estimate flipped the semi join. Equality on decimals also got bad estimates, since interval analysis counts every scale step: TPC-DS `ca_gmt_offset = -5` on `DECIMAL(5, 2)` was estimated at 100 rows instead of 18,246. ## What changes are included in this PR? - Commit 1 is apache#25790 unchanged. - When a column in the predicate has an unknown number of values (no min/max, or a range too wide to count), `analyze` returns no selectivity and `FilterExec` uses the default. This applies to every type, so it also changes estimates on main, e.g. `HAVING count(*) > 10` goes from 100% to 20%. - Casts that bounds can't propagate back through (e.g. decimal to `DOUBLE`) aren't analyzed. Integer to decimal casts that keep every value now propagate bounds, using one helper shared with `builder.rs`. - Decimal equality keeps the estimate it had before apache#25790. ## What is the testing strategy for this PR? - Unit tests for each case, plus `filter_statistics_decimal.slt` with a Q18 plan check and the cases from review. - All 22 TPC-H plans (SF1, SF10) match main, with `pushdown_filters` on and off. - TPC-DS (SF1): q8, q34, q70 and q73 change. They filter on `count(*)` or `rank()` outputs, which have no min/max and were estimated at 100% on main. Locally they ran the same or faster: | Query | main | This PR | | ----- | --------------- | --------------- | | q8 | 0.49 s, 123 MB | 0.50 s, 124 MB | | q34 | 0.18 s, 102 MB | 0.13 s, 85 MB | | q70 | 1.52 s, 173 MB | 1.11 s, 128 MB | | q73 | 0.17 s, 102 MB | 0.13 s, 85 MB | - The Q6-style query from apache#25613 still improves: 111,291 estimated rows (actual 114,160) vs 1,200,243 on main. ## Are there any user-facing changes? Better estimates for range filters on decimal columns with min/max stats. Filters on columns without min/max now use the default selectivity instead of 100%, for any type. One new public function each: `check_statistics_support` and `is_lossless_decimal_cast`. `analyze` is public and now returns `selectivity: None` when a column in the predicate has an unknown number of values (e.g. no min/max), where it used to return `Some(1.0)`. Callers should fall back to a default selectivity, as `FilterExec` does. --------- Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
On TPC-H Q18, DataFusion 56 takes about 233 s and 22 GB of peak memory on an M3 Max, compared with about 7 s and 4 GB on 55.1.0. On x86_64 the process runs out of memory (#27108). @devanbenz traced the regression to #25790: the new Decimal filter selectivity estimate changes the join order of the plan.
As agreed in #24461 (comment), this PR reverts #25790 before the 56 release.
#27146 tracks the work to re-land the change.
What changes are included in this PR?
This reverts commit 6145a09. The only conflict was
filter_statistics_decimal.slt, which #25790 added and #26761 later edited. This PR deletes it.What is the testing strategy for this PR?
The existing tests, which pass locally.
Are there any user-facing changes?
Filters on Decimal columns use the selectivity estimate from 55.x again.
Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.