Skip to content

Create a pipeline/morsel scheduler #21719

Open

Description

@Dandandan

Is your feature request related to a problem or challenge?

Currently we execute the physical plan using mostly a Volcano model.

Scans also currently (mostly)is executed with local aggregation state as each partition is spawned on the tokio runtime (and we usually use the number of threads by default).

We don't control thread locality everywhere, as we spawn tasks for each partition, tokio is free to move/steal them which might increase cross-thread movement.

Describe the solution you'd like

Implement a morsel/pipeline-driven scheduler:

#2226 implemented this before but was removed after some time, probably it didn't show enough traction at that time.

Describe alternatives you've considered

Just keep/improve the current execution model and make it work better within the multi-threaded Tokio runtime.

Additional context

tokiouring seems like it could work very well together .

Activity

  1. RatulDawar commented on Apr 19, 2026

    @RatulDawar
    Contributor

    Hey @Dandandan, is there any task that needs help here ? I can put in my bandwidth to explore this. Seems really interesting to me.

  2. alamb commented on Apr 20, 2026

    @alamb
    Contributor

    I am not sure that most plans suffer from tokio rescheduling them

    I think the biggest improvement would be to avoid having to move IO across cores (aka do IO and then read it , rather than stalling having to wait for IO from some other thread/core on a blocking thread)

  3. Dandandan commented on Apr 20, 2026

    @Dandandan
    ContributorAuthor

    It probably needs some metric to see how often a thread is being moved beyond cores with the current approach.

    I think with some metric/example to showcase this in different scenario's would help first, perhaps simple scan+filter+aggregate/joins queries (like Clickbench) have very good thread locality "by accident" as the tasks probably will stick to a single thread anyway, but perhaps in more nested scenario's (e.g. concurrent partitions > cores) I feel like there might be suboptimal cases where we just spawn too much tasks at once.

    But perhaps it doesn't need "morsel scheduler" per se for that, just better pipelining.

    I think the biggest improvement would be to avoid having to move IO across cores (aka do IO and then read it , rather than stalling having to wait for IO from some other thread/core on a blocking thread)

    Yes, for scan performance, agreed! (Not sure if it needs io_uring or we could do something without?)

  4. Dandandan commented on Apr 20, 2026

    @Dandandan
    ContributorAuthor

    So @RatulDawar perhaps a showcase where thread/context switches happen in which benchmarks would be useful.

    We could be just measuring the switches in thread id for operators with the same partition (e.g. aggregate / sort / join).

    I suspect with queries with Union / SortPreservingMergeExec and similar operators, there could be lessons from that (or show that we are optimal already).

  5. RatulDawar commented on Apr 22, 2026

    @RatulDawar
    Contributor

    @Dandandan so I have been trying certain things for past 2 days and wanted to share here so that you can review my approach.
    I wanted to first try profilers like tokio-console, which I tried setting up locally but was unable to stream datafusion metrics to it.

    So I switched to logging thead swtich events and ran a short LLM generated experiment to simulate the same. Here are the details so you can take a brief look.

    If the logging approach looks correct I can bundle datafusion with something similar to this and run benchmarks to get these metrics per benchmark.
    Let me know that you thing about here.

  6. retroaktive commented on Jul 17, 2026

    @retroaktive

    A data point for this discussion, from a mixed-workload fairness measurement we ran on DataFusion 52.5's stock tokio-stream execution (multi-threaded runtime, operators unmodified).

    Setup: open-loop generator (arrivals don't wait for completions), 1,000 QPS of small partition-pruned point queries, first isolated, then mixed with 5 concurrent heavy full-scan aggregations, 30 s per phase, two back-to-back runs to check reproducibility (headline metrics repeated within ±5 %). Hardware: GCP c3-standard-22 (22 vCPU), OS background services quiesced, data on local NVMe.

    Result: the tail shape stays healthy under contention — p99/p95 ≈ 1.06 in both phases — but the whole small-query latency distribution shifts up when the heavy scans run: p95 inflates 1.38–1.40× vs isolated (1.86–1.95 ms → 2.60–2.69 ms), consistent across both runs. No queries shed or failed in either phase.

    The failure mode looks like the one discussed here: with pull-based streams on a shared runtime, heavy operators get scheduling weight proportional to their poll appetite, and there is no mechanism to hold interactive latency steady against them short of running separate runtimes/pools per class. The absolute numbers are small (sub-3 ms), but for latency-SLO'd interactive workloads the ~1.4× multiplier is workload-structural, not noise — it tracks the heavy queries' presence, not load level.

    We had pre-registered ≤1.3× as our acceptable inflation bound before running (against the morsel-driven-scheduler literature's fairness claims), so for our use case this measurement tipped the decision toward building a morsel-style driver above DataFusion's operators (operators unchanged — only the stream driver replaced, roughly the seam the 2022 scheduler explored). Happy to share more methodology details if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions