Skip to content

Revert #25790 (Decimal filter selectivity) due to TPC-H Q18 regression - #27145

Merged
alamb merged 1 commit into
apache:mainfrom
asolimando:asolimando/revert-25790
Nov 5, 2026
Merged

alamb merged 1 commit into
apache:mainfrom
asolimando:asolimando/revert-25790

Conversation

@asolimando

@asolimando asolimando commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

On TPC-H Q18, DataFusion 56 takes about 233 s and 22 GB of peak memory on an M3 Max, compared with about 7 s and 4 GB on 55.1.0. On x86_64 the process runs out of memory (#27108). @devanbenz traced the regression to #25790: the new Decimal filter selectivity estimate changes the join order of the plan.

As agreed in #24461 (comment), this PR reverts #25790 before the 56 release.

#27146 tracks the work to re-land the change.

What changes are included in this PR?

This reverts commit 6145a09. The only conflict was filter_statistics_decimal.slt, which #25790 added and #26761 later edited. This PR deletes it.

What is the testing strategy for this PR?

The existing tests, which pass locally.

Are there any user-facing changes?

Filters on Decimal columns use the selectivity estimate from 55.x again.


Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

@devanbenz

devanbenz commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Ope, appears great minds think alike. I've opened a PR for this as well 😆

I'll go ahead and close mine.

@alamb alamb left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you -- I verified this is a clean revert.

Thank you @asolimando and @devanbenz

@alamb
alamb enabled auto-merge November 5, 2026 21:00
@alamb
alamb added this pull request to the merge queue Nov 5, 2026
Merged via the queue into apache:main with commit 8460f4f Nov 5, 2026
44 of 46 checks passed
mbutrovich pushed a commit that referenced this pull request Nov 6, 2026
…18 regression (#27145) (#27147)

- Part of #24461

This PR:
- Backports #27145 to the `branch-56` line, to fix the TPC-H Q18
regression in #27108
bvolpato pushed a commit to bvolpato/datafusion that referenced this pull request Nov 21, 2026
…vity for columns without known bounds (apache#27148)

## Which issue does this PR close?

- Closes apache#27146.
- Re-lands apache#25790 (reverted in apache#27145 because of apache#27108).

## Rationale for this change

This PR does two things:
1. Re-lands apache#25790, which estimates decimal filters with interval
analysis.
2. For any type, not just decimals, uses the default selectivity when
interval
analysis can't estimate a filter, e.g. on a column without min/max such
as
a `count(*)` or `rank()` output. These filters were estimated at 100% of
   rows before.

apache#25790 let interval analysis estimate decimal filters. When interval
analysis
can't tell how many values of a column pass, it estimated 100% of rows,
which
is worse than the 20% default. Q18's `HAVING sum(l_quantity) > 300` hit
this:
the aggregate output has no min/max, and the bad estimate flipped the
semi
join.

Equality on decimals also got bad estimates, since interval analysis
counts
every scale step: TPC-DS `ca_gmt_offset = -5` on `DECIMAL(5, 2)` was
estimated
at 100 rows instead of 18,246.

## What changes are included in this PR?

- Commit 1 is apache#25790 unchanged.
- When a column in the predicate has an unknown number of values (no
min/max,
  or a range too wide to count), `analyze` returns no selectivity and
  `FilterExec` uses the default. This applies to every type, so it also
changes estimates on main, e.g. `HAVING count(*) > 10` goes from 100% to
  20%.
- Casts that bounds can't propagate back through (e.g. decimal to
`DOUBLE`)
  aren't analyzed. Integer to decimal casts that keep every value now
  propagate bounds, using one helper shared with `builder.rs`.
- Decimal equality keeps the estimate it had before apache#25790.

## What is the testing strategy for this PR?

- Unit tests for each case, plus `filter_statistics_decimal.slt` with a
Q18
  plan check and the cases from review.
- All 22 TPC-H plans (SF1, SF10) match main, with `pushdown_filters` on
and
  off.
- TPC-DS (SF1): q8, q34, q70 and q73 change. They filter on `count(*)`
or
`rank()` outputs, which have no min/max and were estimated at 100% on
main.
  Locally they ran the same or faster:

  | Query | main            | This PR         |
  | ----- | --------------- | --------------- |
  | q8    | 0.49 s, 123 MB  | 0.50 s, 124 MB  |
  | q34   | 0.18 s, 102 MB  | 0.13 s, 85 MB   |
  | q70   | 1.52 s, 173 MB  | 1.11 s, 128 MB  |
  | q73   | 0.17 s, 102 MB  | 0.13 s, 85 MB   |

- The Q6-style query from apache#25613 still improves: 111,291 estimated rows
  (actual 114,160) vs 1,200,243 on main.

## Are there any user-facing changes?

Better estimates for range filters on decimal columns with min/max
stats.
Filters on columns without min/max now use the default selectivity
instead of
100%, for any type. One new public function each:
`check_statistics_support`
and `is_lossless_decimal_cast`.


`analyze` is public and now returns `selectivity: None` when a column in
the
predicate has an unknown number of values (e.g. no min/max), where it
used to
return `Some(1.0)`. Callers should fall back to a default selectivity,
as
`FilterExec` does.

---------

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TPC-H Q18 regression

3 participants