Skip to content

Prove predicates true from file statistics and cache stats rewrites - #10686

Draft
robert3005 wants to merge 2 commits into
rk/exact-key-probesfrom
rk/stats-satisfy
Draft

robert3005 wants to merge 2 commits into
rk/exact-key-probesfrom
rk/stats-satisfy

Conversation

@robert3005

@robert3005 robert3005 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Stats rewrites could only falsify a predicate. A scan can also skip a predicate that statistics prove true for every row, for example a hash join bound that covers the whole range of a column in a file. Binary comparisons, AND, OR and BETWEEN now get satisfy rules: a comparison is proven when the bounds of its operands order every pair of values and neither operand holds a null or a NaN.

A scan binds its filter once against the table schema and proves it from the statistics of each file, so the same predicate was rewritten into the same proofs for every file. The session now caches falsify and satisfy rewrites, keyed by the identity of the predicate tree with WeakBoundExprKey, so a cached entry does not keep a finished scan's filter alive. Registering a rule clears the cache. Predicates that contain a dynamic expression are not cached, because their proofs hold the session. Proving a filter over 64 files took 36–53% less time for falsify and 21–29% less for satisfy, from 8 to 2048 columns, in a local benchmark that is not part of this PR.

Stacked on #10615.

🤖 Generated with Claude Code

@codspeed

codspeed Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 16.9%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 65 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 5 improved benchmarks
❌ 9 regressed benchmarks
✅ 2394 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime filter_neon[8] 915 ns 2,032 ns -54.97%
❌ Walltime threshold_neon[8] 1.4 µs 2.5 µs -45.14%
❌ Walltime filter_neon[64] 1.8 µs 2.7 µs -33.82%
❌ Walltime threshold_neon[80] 3.6 µs 5.1 µs -29.81%
❌ Walltime filter_neon[80] 2.5 µs 3.5 µs -28.51%
❌ Walltime threshold_neon[64] 2.6 µs 3.5 µs -25.73%
❌ Walltime scalar_subtract_neon 11.2 µs 12.9 µs -13.22%
❌ Walltime decode_chunked_neon[i64, 524288] 12.8 µs 14.4 µs -11.05%
❌ Walltime starts_with_avx2[64, VarBin] 56.9 µs 63.5 µs -10.39%
⚡ CPU Simulation test_scan_filter_to_arrow_array[int_float] 4.1 ms 3.5 ms +15.89%
⚡ CPU Simulation test_repeated_scan_filter[int_float] 3.9 ms 3.4 ms +12.42%
⚡ CPU Simulation test_repeated_scan_filter[int] 2.7 ms 2.4 ms +11.48%
⚡ CPU Simulation test_scan_filter_to_arrow_array[int] 2.8 ms 2.5 ms +11.15%
⚡ CPU Simulation test_scan_filter[int] 3 ms 2.8 ms +10.05%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/stats-satisfy (e5d3cc8) with develop (1dba73c)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005
robert3005 marked this pull request as draft October 23, 2026 16:51
@robert3005

Copy link
Copy Markdown
Contributor Author

Let's merge this after df dynamic expression

@connortsui20 connortsui20 mentioned this pull request Nov 2, 2026
19 of 57 tasks
connortsui20 added a commit that referenced this pull request Nov 3, 2026
…11114

Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 15, 2026
@robert3005 robert3005 removed the stale This PR is stale and will be auto-closed soon label Nov 15, 2026
Stats rewrites could only falsify a predicate. A scan can also skip a
predicate that statistics prove true for every row, for example a hash
join bound that covers the whole range of a column in a file.

Binary comparisons, AND, OR and BETWEEN get `satisfy` rules. A
comparison is proven when the bounds of its operands order every pair
of values and neither operand holds a null or a NaN.

Signed-off-by: Robert Kruszewski <github@robertk.io>
A scan binds its filter once against the table schema and proves it
from the statistics of each file, so it rewrote the same predicate into
the same proofs once for every file.

The session now caches the falsify and satisfy rewrites of a predicate
by the identity of its tree. A key does not keep its predicate alive,
and an insert drops the entries of predicates that no scan holds. A
rule registration clears the cache, and a rewrite that started before
it is not kept. Predicates with a dynamic expression are not cached,
because their proofs hold the session.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 changed the title Prove comparisons true for every row from statistics Prove predicates true from file statistics and cache stats rewrites Nov 19, 2026
@robert3005
robert3005 changed the base branch from develop to rk/exact-key-probes November 19, 2026 15:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants