Repository navigation
Conversation
Added short circuiting extreme values check on list.
Added more tests to boolean InList due to abundance of special cases in implementation.
Codecov Report
@@ Coverage Diff @@
## master #278 +/- ##
==========================================
- Coverage 76.80% 76.06% -0.75%
==========================================
Files 133 140 +7
Lines 23284 23736 +452
==========================================
+ Hits 17884 18055 +171
- Misses 5400 5681 +281
Continue to review full report at Codecov.
|
|
Cool @pjmore ! |
|
After doing more benchmarking there's more performance degradation for small lists than performance gains on large ones so I'm closing the pr for now. |
|
@pjmore I think for this optimization to make sense, it needs a certain threshold for where it moves to the different implementation (e.g. more than 500 values (or whenever it is faster) -> use hash set ). |
* Use DataFusion 20 * update lockfile
Added short circuiting extreme values check on list.
Which issue does this PR close?
fixes #145
What changes are included in this PR?
Various changes to the InList physical expression.
Macro based solution was replaced with generic functions, which ended up being a little symbol soupy. If the macro based solution is preferred I'd be happy to switch it to use a macro instead.
List of optimizations:
The thresholds for selecting whether to do a scan or use a HashSet could be tuned more.
Are there any user-facing changes?
No