Closed
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster
Description
Is your feature request related to a problem or challenge?
I ran some benchmarks in constrained Docker containers and found that DataFusion is pretty close to DuckDB speed when running on a single core but does not scale as well as DuckDB when more cores are added.
At 16 cores, DuckDB was ~2.9x faster than DataFusion for this particular test.
| Cores | DuckDB 0.7.1 | DataFusion Python 21.0.0 | DuckDB x Faster |
|---|---|---|---|
| 1 | 315,630.9 | 318,037.8 | 1.0 |
| 2 | 155,034.5 | 197,257.3 | 1.3 |
| 4 | 75,110.0 | 111,243.1 | 1.5 |
| 8 | 37,584.9 | 73,142.3 | 1.9 |
| 16 | 18,880.3 | 55,071.8 | 2.9 |
Describe the solution you'd like
I would like DataFusion to scale as well as DuckDB as more cores are added.
Describe alternatives you've considered
No response
Additional context
Instructions for creating the Docker images can be found at https://github.com/sql-benchmarks/sqlbench-runners
Reacted by Rich, dudzicp, r.4ntix and Austin Liu
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster
Activity
alamb commentedon Apr 14, 2023
Possibly related: #5995
Dandandan commentedon Apr 14, 2023
Suggestion I posted in slack:
I wonder whether there might be some regressions here wrt scalability with the nr of cores - #4219 comes to mind which makes building the hashmap for the build-side done in a single thread for smaller tables
might be checked by changing datafusion.optimizer.hash_join_single_partition_threshold
Dandandan commentedon Apr 14, 2023
For tpch-h q1 (SF=1) the metrics are as follows for the complete plan with metrics is as follows:
Relevant metrics:
andygrove commentedon Apr 17, 2023
I ran the benchmarks with a simpler version of q1 with just the table scan and filter:
Here are the results:
andygrove commentedon Apr 17, 2023
Here are the results for just the table scan:
This screenshot shows CPU usage with the DataFusion run on the left and the DuckDB run on the right, both using 32 cores.
Here is the directory listing of my lineitem directory:
andygrove commentedon Apr 17, 2023
@Dandandan I added more benchmark results. The issues appear to be related to the
ParquetExec.Dandandan commentedon Apr 18, 2023
I could also replicate the issue of non-perfect scaling with loading the tables in memory.
One thing I noticed is that the current round-robin
RepartitionExecdoesn't spread the batches evenly over the number of output channels, which can already be seen in MemoryExec itself:MemoryExec: partitions=32, partition_sizes=[32, 32, 32, 32, 32, 32, 32, 32, 26, 26, 26, 25, 25, 25, 25, 25, 25, 25, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], metrics=[]It has a bias for the starting partitions as the outputs always go first to those channels.
Dandandan commentedon Apr 18, 2023
Also, I wonder if
DistributionSender/DistributionReceiverdidn't introduce any regressions here FYI @crepererumcrepererum commentedon Apr 18, 2023
Definitely possible, even if it's only due to changes it triggers in tokio's scheduler. On the other hand, tokio's scheduler isn't really optimized for the DataFusion use case in the first place (as @tustvold probably pointed out before).
cristian-ilies-vasile commentedon Apr 20, 2023
There is a small application written in Rust called odbc2parquet that reads data from databases and exports parquet files.
For testing on dummy data I think it is useful.
https://github.com/pacman82/odbc2parquet
andygrove commentedon Apr 25, 2023
Here are latest results with DataFusion Python 23, which fixes a bug around how the Tokio runtime was being managed.
Dandandan commentedon Apr 25, 2023
Interestingly, the speedup seems bigger for fewer cores.
There have been quite a few improvement for Q1 between 21 and 23 for aggregates and casts, that might explain most of the difference.
alamb commentedon Jul 12, 2023
I believe #6929 from @YjyJeff will perhaps help with this
andygrove commentedon Nov 27, 2023
Latest results with DF 33
alamb commentedon Nov 27, 2023
I believe that #6937 (basically that the fact we are doing much more work as the cores go up (as each group gets processed in each core) contributes to this problem
andygrove commentedon Nov 27, 2023
andygrove commentedon Nov 27, 2023
I produced the chart using scripts at https://github.com/sql-benchmarks/sqlbench-runners/blob/main/scripts
comphead commentedon Nov 27, 2023
@sunchao @viirya cc ^^^^
andygrove commentedon Nov 27, 2023
The regression between 32 and 33 seems to be caused by my "improvement" to filter statistics, although we saw that others saw big improvements with 33 (such as https://x.com/mim_djo/status/1725510410567307702?s=46&t=8JleG5FR5SAVU1vl35CPLg) so maybe this is environment specific. I will continue to debug this in the next few days.
Dandandan commentedon Nov 28, 2023
I wonder if this is just bad configuration?
I created a PR to update the DF configurations to use the defaults.
datafusion-contrib/sqlbench-runners#31
andygrove commentedon Nov 28, 2023
@Dandandan was correct. It was the configs that were the issue. Here is a run of just versions 32 and 33 using default configs.