Skip to content

In list performance improvements - #278

Closed
pjmore wants to merge 9 commits into
apache:masterfrom
pjmore:in-list-perf
Closed

pjmore wants to merge 9 commits into
apache:masterfrom
pjmore:in-list-perf

Conversation

@pjmore

@pjmore pjmore commented May 6, 2021

Copy link
Copy Markdown
Contributor

Added short circuiting extreme values check on list.

Which issue does this PR close?

fixes #145

What changes are included in this PR?

Various changes to the InList physical expression.

Macro based solution was replaced with generic functions, which ended up being a little symbol soupy. If the macro based solution is preferred I'd be happy to switch it to use a macro instead.

List of optimizations:

  • Using a HashSet to check for values in the list for large lists
  • For numeric lists sort the list of values:
    • Check that value falls within extreme values of list to enable
    • Short circuits the linear search
  • Explicitly lift check for negated and contains_null to outside of the loop. It probably doesn't matter and if it did the compiler probably already does this, but I thought it made sense to be explicit.

The thresholds for selecting whether to do a scan or use a HashSet could be tuned more.

Are there any user-facing changes?

No

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

Merging #278 (c052c14) into master (e271e4d) will decrease coverage by 0.74%.
The diff coverage is 59.31%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #278      +/-   ##
==========================================
- Coverage   76.80%   76.06%   -0.75%     
==========================================
  Files         133      140       +7     
  Lines       23284    23736     +452     
==========================================
+ Hits        17884    18055     +171     
- Misses       5400     5681     +281     
Impacted Files Coverage Δ
...atafusion/src/physical_plan/expressions/in_list.rs 68.52% <59.31%> (-9.87%) ⬇️
datafusion/src/physical_plan/coalesce_batches.rs 83.18% <0.00%> (-1.77%) ⬇️
...tafusion/src/physical_plan/distinct_expressions.rs 90.34% <0.00%> (-0.57%) ⬇️
datafusion/src/datasource/csv.rs 73.07% <0.00%> (-0.26%) ⬇️
...lista/rust/core/src/serde/logical_plan/to_proto.rs 68.05% <0.00%> (-0.10%) ⬇️
datafusion/src/physical_plan/csv.rs 79.01% <0.00%> (-0.09%) ⬇️
datafusion/src/physical_plan/functions.rs 89.42% <0.00%> (-0.06%) ⬇️
...sta/rust/core/src/serde/logical_plan/from_proto.rs 40.51% <0.00%> (-0.06%) ⬇️
datafusion/src/sql/planner.rs 83.62% <0.00%> (-0.03%) ⬇️
datafusion/src/physical_plan/parquet.rs 87.55% <0.00%> (-0.02%) ⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e271e4d...c052c14. Read the comment docs.

@Dandandan

Copy link
Copy Markdown
Contributor

Cool @pjmore !
I think these kind of improvements could really benefit from some (micro) benchmarks - do you feel like adding that in this PR so we can see the change in performance?

@pjmore

pjmore commented May 7, 2021 •

Copy link
Copy Markdown
Contributor Author

After doing more benchmarking there's more performance degradation for small lists than performance gains on large ones so I'm closing the pr for now.

@pjmore pjmore closed this May 7, 2021
@Dandandan

Copy link
Copy Markdown
Contributor

@pjmore I think for this optimization to make sense, it needs a certain threshold for where it moves to the different implementation (e.g. more than 500 values (or whenever it is faster) -> use hash set ).

HairstonE pushed a commit to HairstonE/datafusion that referenced this pull request Oct 7, 2025
* Use DataFusion 20

* update lockfile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve performance if IN list function

3 participants