You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge?
Currently we execute the physical plan using mostly a Volcano model.
Scans also currently (mostly)is executed with local aggregation state as each partition is spawned on the tokio runtime (and we usually use the number of threads by default).
We don't control thread locality everywhere, as we spawn tasks for each partition, tokio is free to move/steal them which might increase cross-thread movement.
Describe the solution you'd like
Implement a morsel/pipeline-driven scheduler:
#2226 implemented this before but was removed after some time, probably it didn't show enough traction at that time.
Describe alternatives you've considered
Just keep/improve the current execution model and make it work better within the multi-threaded Tokio runtime.
Additional context
tokiouring seems like it could work very well together .
I am not sure that most plans suffer from tokio rescheduling them
I think the biggest improvement would be to avoid having to move IO across cores (aka do IO and then read it , rather than stalling having to wait for IO from some other thread/core on a blocking thread)
It probably needs some metric to see how often a thread is being moved beyond cores with the current approach.
I think with some metric/example to showcase this in different scenario's would help first, perhaps simple scan+filter+aggregate/joins queries (like Clickbench) have very good thread locality "by accident" as the tasks probably will stick to a single thread anyway, but perhaps in more nested scenario's (e.g. concurrent partitions > cores) I feel like there might be suboptimal cases where we just spawn too much tasks at once.
But perhaps it doesn't need "morsel scheduler" per se for that, just better pipelining.
I think the biggest improvement would be to avoid having to move IO across cores (aka do IO and then read it , rather than stalling having to wait for IO from some other thread/core on a blocking thread)
Yes, for scan performance, agreed! (Not sure if it needs io_uring or we could do something without?)
So @RatulDawar perhaps a showcase where thread/context switches happen in which benchmarks would be useful.
We could be just measuring the switches in thread id for operators with the same partition (e.g. aggregate / sort / join).
I suspect with queries with Union / SortPreservingMergeExec and similar operators, there could be lessons from that (or show that we are optimal already).
@Dandandan so I have been trying certain things for past 2 days and wanted to share here so that you can review my approach.
I wanted to first try profilers like tokio-console, which I tried setting up locally but was unable to stream datafusion metrics to it.
So I switched to logging thead swtich events and ran a short LLM generated experiment to simulate the same. Here are the details so you can take a brief look.
If the logging approach looks correct I can bundle datafusion with something similar to this and run benchmarks to get these metrics per benchmark.
Let me know that you thing about here.
A data point for this discussion, from a mixed-workload fairness measurement we ran on DataFusion 52.5's stock tokio-stream execution (multi-threaded runtime, operators unmodified).
Setup: open-loop generator (arrivals don't wait for completions), 1,000 QPS of small partition-pruned point queries, first isolated, then mixed with 5 concurrent heavy full-scan aggregations, 30 s per phase, two back-to-back runs to check reproducibility (headline metrics repeated within ±5 %). Hardware: GCP c3-standard-22 (22 vCPU), OS background services quiesced, data on local NVMe.
Result: the tail shape stays healthy under contention — p99/p95 ≈ 1.06 in both phases — but the whole small-query latency distribution shifts up when the heavy scans run: p95 inflates 1.38–1.40× vs isolated (1.86–1.95 ms → 2.60–2.69 ms), consistent across both runs. No queries shed or failed in either phase.
The failure mode looks like the one discussed here: with pull-based streams on a shared runtime, heavy operators get scheduling weight proportional to their poll appetite, and there is no mechanism to hold interactive latency steady against them short of running separate runtimes/pools per class. The absolute numbers are small (sub-3 ms), but for latency-SLO'd interactive workloads the ~1.4× multiplier is workload-structural, not noise — it tracks the heavy queries' presence, not load level.
We had pre-registered ≤1.3× as our acceptable inflation bound before running (against the morsel-driven-scheduler literature's fairness claims), so for our use case this measurement tipped the decision toward building a morsel-style driver above DataFusion's operators (operators unchanged — only the stream driver replaced, roughly the seam the 2022 scheduler explored). Happy to share more methodology details if useful.
Activity
RatulDawar commentedon Apr 19, 2026
Hey @Dandandan, is there any task that needs help here ? I can put in my bandwidth to explore this. Seems really interesting to me.
alamb commentedon Apr 20, 2026
I am not sure that most plans suffer from tokio rescheduling them
I think the biggest improvement would be to avoid having to move IO across cores (aka do IO and then read it , rather than stalling having to wait for IO from some other thread/core on a blocking thread)
Dandandan commentedon Apr 20, 2026
It probably needs some metric to see how often a thread is being moved beyond cores with the current approach.
I think with some metric/example to showcase this in different scenario's would help first, perhaps simple scan+filter+aggregate/joins queries (like Clickbench) have very good thread locality "by accident" as the tasks probably will stick to a single thread anyway, but perhaps in more nested scenario's (e.g. concurrent partitions > cores) I feel like there might be suboptimal cases where we just spawn too much tasks at once.
But perhaps it doesn't need "morsel scheduler" per se for that, just better pipelining.
Yes, for scan performance, agreed! (Not sure if it needs io_uring or we could do something without?)
Dandandan commentedon Apr 20, 2026
So @RatulDawar perhaps a showcase where thread/context switches happen in which benchmarks would be useful.
We could be just measuring the switches in thread id for operators with the same partition (e.g. aggregate / sort / join).
I suspect with queries with Union / SortPreservingMergeExec and similar operators, there could be lessons from that (or show that we are optimal already).
RatulDawar commentedon Apr 22, 2026
@Dandandan so I have been trying certain things for past 2 days and wanted to share here so that you can review my approach.
I wanted to first try profilers like tokio-console, which I tried setting up locally but was unable to stream datafusion metrics to it.
So I switched to logging thead swtich events and ran a short LLM generated experiment to simulate the same. Here are the details so you can take a brief look.
If the logging approach looks correct I can bundle datafusion with something similar to this and run benchmarks to get these metrics per benchmark.
Let me know that you thing about here.
retroaktive commentedon Jul 17, 2026
A data point for this discussion, from a mixed-workload fairness measurement we ran on DataFusion 52.5's stock tokio-stream execution (multi-threaded runtime, operators unmodified).
Setup: open-loop generator (arrivals don't wait for completions), 1,000 QPS of small partition-pruned point queries, first isolated, then mixed with 5 concurrent heavy full-scan aggregations, 30 s per phase, two back-to-back runs to check reproducibility (headline metrics repeated within ±5 %). Hardware: GCP c3-standard-22 (22 vCPU), OS background services quiesced, data on local NVMe.
Result: the tail shape stays healthy under contention — p99/p95 ≈ 1.06 in both phases — but the whole small-query latency distribution shifts up when the heavy scans run: p95 inflates 1.38–1.40× vs isolated (1.86–1.95 ms → 2.60–2.69 ms), consistent across both runs. No queries shed or failed in either phase.
The failure mode looks like the one discussed here: with pull-based streams on a shared runtime, heavy operators get scheduling weight proportional to their poll appetite, and there is no mechanism to hold interactive latency steady against them short of running separate runtimes/pools per class. The absolute numbers are small (sub-3 ms), but for latency-SLO'd interactive workloads the ~1.4× multiplier is workload-structural, not noise — it tracks the heavy queries' presence, not load level.
We had pre-registered ≤1.3× as our acceptable inflation bound before running (against the morsel-driven-scheduler literature's fairness claims), so for our use case this measurement tipped the decision toward building a morsel-style driver above DataFusion's operators (operators unchanged — only the stream driver replaced, roughly the seam the 2022 scheduler explored). Happy to share more methodology details if useful.