Repository navigation
Prove predicates true from file statistics and cache stats rewrites - #10686
robert3005 wants to merge 2 commits into
Conversation
Merging this PR will degrade performance by 16.9%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
|
Let's merge this after df dynamic expression |
58d5cc5 to
e5d3cc8
Compare
…11114 Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
Stats rewrites could only falsify a predicate. A scan can also skip a predicate that statistics prove true for every row, for example a hash join bound that covers the whole range of a column in a file. Binary comparisons, AND, OR and BETWEEN get `satisfy` rules. A comparison is proven when the bounds of its operands order every pair of values and neither operand holds a null or a NaN. Signed-off-by: Robert Kruszewski <github@robertk.io>
A scan binds its filter once against the table schema and proves it from the statistics of each file, so it rewrote the same predicate into the same proofs once for every file. The session now caches the falsify and satisfy rewrites of a predicate by the identity of its tree. A key does not keep its predicate alive, and an insert drops the entries of predicates that no scan holds. A rule registration clears the cache, and a rewrite that started before it is not kept. Predicates with a dynamic expression are not cached, because their proofs hold the session. Signed-off-by: Robert Kruszewski <github@robertk.io>
e5d3cc8 to
fdf07df
Compare
Stats rewrites could only falsify a predicate. A scan can also skip a predicate that statistics prove true for every row, for example a hash join bound that covers the whole range of a column in a file. Binary comparisons, AND, OR and BETWEEN now get
satisfyrules: a comparison is proven when the bounds of its operands order every pair of values and neither operand holds a null or a NaN.A scan binds its filter once against the table schema and proves it from the statistics of each file, so the same predicate was rewritten into the same proofs for every file. The session now caches falsify and satisfy rewrites, keyed by the identity of the predicate tree with
WeakBoundExprKey, so a cached entry does not keep a finished scan's filter alive. Registering a rule clears the cache. Predicates that contain a dynamic expression are not cached, because their proofs hold the session. Proving a filter over 64 files took 36–53% less time for falsify and 21–29% less for satisfy, from 8 to 2048 columns, in a local benchmark that is not part of this PR.Stacked on #10615.
🤖 Generated with Claude Code