Skip to content

feat: store start/end bounds in Partitioning::Range - #26593

Open
jayshrivastava wants to merge 12 commits into
apache:mainfrom
jayshrivastava:js/range-partition-bounds
Open

jayshrivastava wants to merge 12 commits into
apache:mainfrom
jayshrivastava:js/range-partition-bounds

Conversation

@jayshrivastava

@jayshrivastava jayshrivastava commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Important

Fork CI setup: For pull requests from a fork to main, enable workflows in your fork's Actions tab before you push.
See the fork CI setup guide.

Which issue does this PR close?

Rationale for this change

Most of the context is in the issue above.

The main use case is that I want to partition a query across multiple machines while assigning Partitioning::Range to each machine. Because we don't store a start and end bound, you lose information:

Partitioning::Range(4, split_points: [10, 20, 30]) 

// Split into 2 workers, assuming target_partitions=2
Range(2, split_points: [10]) // 2 partitions: x < 10 and  x >= 10
Range(2, split_points: [30]) // 2 partitions: x < 30, x >= 30

Because split point 20 is lost, both machines have overlapping ranges x >= 10 and x < 30. This means you cannot, for example, coalesce them back into the original partitioning. In that scenario, you you have to basically fall back to Partitioning::Unknown(4)

This change adds optional start and end bounds, so you can split up the range like so:

Partitioning::Range(4, split_points: [10, 20, 30]) 

// Split into 2 workers, assuming target_partitions=2
Partitioning::Range(2, split_points: [10], end_bound: 20) // 2 partitions: x < 10 and  10 <= x < 20 
Partitioning::Range(2, split_points: [30], start_bound: 20) // 2 partitions: 20 <= x < 30 and  30 <= x 

Now, you preserve information.

  • The ranges are partitioned across workers now. Because they are bounded, you can implement certain shuffle optimizations in a Range - Range shuffle.
  • The ranges can be coalesced back into Partitioning::Range(4) instead of Partitioning::Unknown(4)

What changes are included in this PR?

The diff is large but it's a very straightforward, mechanical change.

  1. Partitioning::Range now has 2 new fields.
    /// Inclusive start of the key space, in `ordering` order.
    start_bound: Option<SplitPoint>,
    /// Exclusive end of the key space, in `ordering` order.
    end_bound: Option<SplitPoint>,

None means "unbounded", which is equivalent to the default today. The difference is that now, as a user, I can split this up into N ranges without losing information by storing split points in the bounds.

  1. Minor change: split points no longer need to be strictly increasing. This allows a user to pad empty ranges if they would like.

  2. To support bounds in Partitioning::Range, RepartitionExec has some minor changes to the router.

What is the testing strategy for this PR?

Unit tests and existing tests.

Are there any user-facing changes?

RangePartitioning can now optionally store bounds in addition to internal split points.

pub fn try_with_bounds(
    mut self,
    start_bound: Option<SplitPoint>,
    end_bound: Option<SplitPoint>,
) -> Result<Self>

FFI_RangePartitioning has two new fields. Because this changes the struct's C layout, this is technically an API change.

start_bound: FFI_Option<SVec<WrappedArray>>,
end_bound: FFI_Option<SVec<WrappedArray>>,

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate common Related to common crate proto Related to proto crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate labels Oct 22, 2026
@jayshrivastava jayshrivastava changed the title wip feat: store start/end bounds in Partitioning::Range Oct 22, 2026
@github-actions github-actions Bot removed the optimizer Optimizer rules label Oct 22, 2026
@jayshrivastava
jayshrivastava marked this pull request as ready for review October 22, 2026 19:29
@jayshrivastava

Copy link
Copy Markdown
Contributor Author

👋🏽 @stuhood @saadtajwar @avantgardnerio would any of you be interested in reviewing this? This is related to #22395, which you all have participated in. Please lmk!

@saadtajwar

Copy link
Copy Markdown
Contributor

👋🏽 @stuhood @saadtajwar @avantgardnerio would any of you be interested in reviewing this? This is related to #22395, which you all have participated in. Please lmk!

Hey @jayshrivastava - thanks for tagging, happy to review this! I'll try to get to this in the next couple of days :)

@jayshrivastava

Copy link
Copy Markdown
Contributor Author

Thanks @saadtajwar!

@saadtajwar saadtajwar left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just a quick question and small suggestion/nit! Super exciting feature :)

Comment on lines +69 to +89
/// according to the provided sort options. Equal split points imply empty partitions.
pub fn validate_range_split_points(
split_points: &[SplitPoint],
sort_options: &[SortOptions],
) -> Result<()> {
let width = sort_options.len();
for (idx, split_point) in split_points.iter().enumerate() {
let split_point_width = split_point.values().len();
if split_point_width != width {
return _plan_err!(
"Range partitioning split point {idx} has width {split_point_width}, but ordering has width {width}"
);
}
}

for (idx, split_points) in split_points.windows(2).enumerate() {
if compare_rows(
split_points[0].values(),
split_points[1].values(),
sort_options,
)? != Ordering::Less
)? == Ordering::Greater

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm out of curiosity, can I ask if there's context around the addition of equal split points? Is there a case for distributed execution in which it would be helpful to have an empty partition that I may be missing?

Another thing which might be useful is to be able to pad empty partitions. Ex. use duplicate split points such as 10, 10, 10, 10 to indicate empty partitions.

I see the above comment in #26509, but would definitely like to understand use cases for the empty partitions :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason is that padding empty partitions is supported Partitioning::Unknown. For example, I I can add an empty file group. Doing the same when a data source is Partitioning::Range introduces an error.

In distributed-datafusion, this feature ensure all tasks (horizontally distributed parts of the plan) have the same number of partitions. This simplifies the code a lot. I imagine the same is true for ballista etc but I'm not 100% sure. We could move away from this if anyone feels strongly, but I figured it's a pretty inoffensive change to make in this repo.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, makes sense to me! Thanks!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In distributed-datafusion, this feature ensure all tasks (horizontally distributed parts of the plan) have the same number of partitions.

We could move away from this if anyone feels strongly, but I figured it's a pretty inoffensive change to make in this repo.

I'd rather DataFusion not take on that invariant without more discussion. As far as I can find, it doesn't have one today. Partition counts already differ from operator to operator, and the only places that need equal counts are children that must be co-partitioned (compatible_co_partitioning_layout) and interleaved inputs (can_interleave). I'd like to keep DataFusion open to running operators with different partition counts, for example an operator that doesn't need as many partitions coalescing adjacent ranges (the SourceFiner case in #25483). Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ.

Bounds already let a task have fewer partitions without padding. In Example 4 of datafusion-contrib/datafusion-distributed#782, the last task could be Range(2, [60], start=50) instead of split points [50, 60] with start=50 and an empty first partition. Could we keep validate_range_split_points strict, require each bound to lie strictly outside the retained samples, and leave padding to DFD? That also removes the empty range where start_bound == end_bound. Once DataFusion accepts equal split points, tightening the check later would reject layouts that downstream code already builds. If the check stays strict, the upgrade-guide entry I asked for on line 69 only needs to cover the bounds fields.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making empty partitions irrepresentable in the Partitioning::Range specification can be cornering.

For example, having an empty FileGroup in a parquet scan is something that can happen, there's no invariant that prevent that, however, that'd automatically make it incompatible with the current specification of Partitioning::Range. At some point, FileScanConfig might want to expose range partitioning here https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/file_scan_config/mod.rs#L908-L940, so how should it react in the event of one of the FileGroups being empty?

One other idea that comes to mind is to explicitly model empty partitions somehow, maybe making the split points accept a ScalarValue::Null and accounting for it?

Regardless of the mechanism, I do see value in making empty partitions representable.

@jayshrivastava jayshrivastava Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partition counts already differ from operator to operator, and the only places that need equal counts are children that must be co-partitioned (compatible_co_partitioning_layout) and interleaved inputs (can_interleave). I'd like to keep DataFusion open to running operators with different partition counts, for example an operator that doesn't need as many partitions coalescing adjacent ranges (the SourceFiner case in #25483). Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ.

I would expect can_interleave and compatible_co_partitioning_layout to be improved so that we are able to interleave arbitrary partitioned data. Ex. Range(3, [60, 80], start=50) should be interleaved with Range(4, [60, 80]). You mentioned this below as well. I'll file an issue.

Also, my experience has been that, if the user wants to avoid a RepartitionExec or wants an InterleaveExec, they have to make sure the correct partitioning properties are declared on their data source. I've even gone as far to declare data sources as Hash() partitioned when they are not. This is to say that I think a user would probably go out of their way to declare interleavable and/or compatible, not necessarily relying on datafusion.

Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ

I'd say the uses for padding are user-defined, like our invariant in DFD and the KLL example Brent mentioned above.

Padding would not really be used to achieve co-partitioning. For example, I don't see a way to pad Range(2, [50]) to make it compatible with Range(3, [50, 60]). Padding with Range(3, [50, 50]) doesn't work. Converting to Range(3, [50, 60]) requires a repartition, splitting up the last partition.

In Example 4 of datafusion-contrib/datafusion-distributed#782, the last task could be Range(2, [60], start=50) instead of split points [50, 60] with start=50 and an empty first partition.

The main distinction is that the former has 2 partitions and latter has 3. You force the user to have an additional tasks spawned if they want to add bounds to Range partitioning.

Padding empty partitions also introduces additional tasks, but the user can opt out of this by simply not padding.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I'd also double check how easy it'd be to remove the invariant in DFD about needing the same partition count in all task-specialized plans in a stage.

If we were able to remove that invariant, will it avoid introducing some sense of padding in this project?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to say thank you for bringing some more eyes and perspectives to this topic!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And thank you for being flexible in your understanding! Always a pleasure...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Appreciate the details reviews from everyone :)

@jayshrivastava

jayshrivastava commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

FAQ

  1. Why make bounds optional instead of using type::MIN and type::MAX

Some types don't have a max or a reasonable value ex. what's the max of a string? "zzzzzz"? For variable length types too

  1. Why is this PR so large?

Tests around range satisfaction. Generally, a join may join two different ranges, with or without bounds. 50% of this PR is just adding test coverage around that. For example, the join type influences the behavior here. Happy to delete tests if people thing any are less useful.

@mbutrovich
mbutrovich self-requested a review November 2, 2026 20:39

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jayshrivastava! The join-type test makes it easy to see which side's bounds each join can inherit.

Should we settle the alternative @stuhood raised in datafusion-contrib/datafusion-distributed#628 (comment) before bounds become part of the public range contract? There, every task keeps the full Range(N, split_points) and the partitions outside its slice are empty, which needs no new fields on RangePartitioning. Spark describes the same kind of slice by index: a coalesced AQE read gives each reader a span of reducer partitions and keeps the original RangePartitioning with a smaller count (AQEShuffleReadExec). As I read your reply in datafusion-contrib/datafusion-distributed#628 (comment) and the padding thread here, this PR still pads tasks with empty partitions, so what the bounds save is the N - N/W idle partitions on each task. In exchange, every range rule has to account for bounds. For example, equality now includes them, so can_interleave rejects inputs that differ only in bounds, and the layout relationships in #25483 will need to define how bounds compose. Is the idle-partition overhead the remaining cost of that approach, or is there a plan shape it can't express?

@jayshrivastava

Copy link
Copy Markdown
Contributor Author

Should we settle the alternative @stuhood raised in datafusion-contrib/datafusion-distributed#628 (comment) before bounds become part of the public range contract? There, every task keeps the full Range(N, split_points) and the partitions outside its slice are empty, which needs no new fields on RangePartitioning.

Stu's approach is valid. I also have a similar one in datafusion-contrib/datafusion-distributed#782. I decided to publish this PR because I felt that those two approaches diverge from our goal of keep DFD as close to DF as possible and as thin of a layer over the single node implementation. Introducing these bounds reduces the complexity of datafusion distributed very significantly, and the lines of code as well.

Ballista opted to implement range shuffles without introducing bounds to DF and you can see the tradeoffs that needed to be made.

Consider that we want to distributed this across two tasks

Global: Range(4, splits=[10, 20, 30])

          p0           p1           p2           p3
     [-inf, 10)    [10, 20)     [20, 30)     [30, +inf)
     \___________________/     \____________________/
             Task 0                    Task 1

In ballista, the RangeShuffleReaderExec reports UnknownPartitioning (see reader properties, task slicing)


                         Range shuffle
                         /           \
                        v             v
                Task 0                 Task 1
           RangeShuffleReader     RangeShuffleReader

  Actual ranges:
           [-inf,10) [10,20)      [20,30) [30,+inf)

  DataFusion property:
           UnknownPartitioning(2) UnknownPartitioning(2)

The UnknownPartitioning makes that within a task, we don't have the range partitioning property anymore. Now, a custom plan node won't know its partitioned and may not be able to make certain optimizations.

Your suggestion is to have each task advertise all partitions but mark some as empty like so

                    p0       p1       p2       p3
  Task 0:           data     data     empty    empty
  Task 1:           empty    empty    data     data

  Both advertise Range(4, splits=[10,20,30]).

That is a reasonable alternative, but those empty partitions have an extra cost, the RepartitionExec spawns a reader task and a completion task for every input partition.

Spark describes the same kind of slice by index: a coalesced AQE read gives each reader a span of reducer partitions and keeps the original RangePartitioning with a smaller count (AQEShuffleReadExec).

On Spark, I think the distinction is that its physical RangePartitioning describes ordering and a partition count; it does not contain concrete split values. You can prove that two tasks have disjoint partitions / ranges because the indexes are disjoint. Ie. one task gets 0,1 and another gets 2,3. In datafusion, you can't prove that partitions are disjoint without bounds.

As I read your reply in datafusion-contrib/datafusion-distributed#628 (comment) and the padding thread here, this PR still pads tasks with empty partitions, so what the bounds save is the N - N/W idle partitions on each task. In exchange, every range rule has to account for bounds. For example, equality now includes them, so can_interleave rejects inputs that differ only in bounds, and the layout relationships in #25483 will need to define how bounds compose. Is the idle-partition overhead the remaining cost of that approach, or is there a plan shape it can't express?

Regarding idle partitions, the user has full control here. They can choose to not have them. This PR only introduces the option to pad.

On can_interleave, I would say the current implementation is correct. We would not interleave these:

Input A:  [0,10) [10,20) [20,30) [30,35)
Input B:  [0,10) [10,20) [20,30) [30,40)

However, we may want to interleave them and use this layout.

Output:   [0,10) [10,20) [20,30) [30,40)

Because this PR is a bit large already, I can file a follow up issue for this. Because bounds are new, I doubt many users will run into this. When someone does, that optimization can be implemented fairly easily.

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jayshrivastava

This looks good, just couple of items.

@mbutrovich

Copy link
Copy Markdown
Contributor

That is a reasonable alternative, but those empty partitions have an extra cost, the RepartitionExec spawns a reader task and a completion task for every input partition.

Thanks, that tells me the difference is run-time cost and DFD code, not a plan the alternative can't express. Since bounds are permanent once they ship, could we measure that cost before settling on them? Each task pays for N - N/W empty partitions: two tasks spawned per input partition in RepartitionExec, plus an empty stream through each operator above the scan. That grows as N per task and N * W across a stage, so a sweep up to the largest shapes DFD expects would show what bounds save. For example, 16 partitions per task on 100 tasks is Range(1600) with 1584 empty partitions on each task. @stuhood also asked in datafusion-contrib/datafusion-distributed#628 (comment) whether a way to declare empty partitions for every Partitioning would remove that cost for Hash too. I don't see that answered yet, and it would avoid a Range-only field.

You can prove that two tasks have disjoint partitions / ranges because the indexes are disjoint. Ie. one task gets 0,1 and another gets 2,3. In datafusion, you can't prove that partitions are disjoint without bounds.

As I read it, the same index argument works in DataFusion under the empty-partition alternative. Every task advertises the same Range(4, [10, 20, 30]), so partition k covers the same keys on every task, and a task that owns 0, 1 has rows only in those partitions. The overlap in the PR description comes from giving each task its own Range(2, [10]) or Range(2, [30]), which drops the global indexes.

I think the more useful comparison with Spark is where it keeps the slice. Its RangePartitioning carries only an ordering and a count. The sampled bounds stay in the exchange's RangePartitioner, and Spark never treats two range layouts as co-partitioned because the bounds are sampled (RangeShuffleSpec). A coalesced read gives each reader a contiguous span of reducer indexes and keeps RangePartitioning with a smaller count (AQEShuffleReadExec). As I read it, that works because a Spark partitioning describes the whole stage, so no task needs to know its own slice.

Spark's coalescing also covers a case DFD will hit: partitions too small to be worth a worker each. AQE packs contiguous reducer partitions up to a target size (ShufflePartitionsUtil). The read keeps RangePartitioning only while every spec is a contiguous, non-overlapping span (isCoalescedRead). Once it splits a skewed partition, it falls back to UnknownPartitioning (L108-L109). In DataFusion terms, coalescing a span drops the split points inside it, so the result is still a plain Range over a subset of the split points. Should DFD coalesce that way before running a range layout on fewer workers? It doesn't remove the per-task slice question, but it shrinks both sides of the trade-off. Under the empty-partition alternative, each task's empty partitions drop from N - N/W to M - M/W for the coalesced count M.

CockroachDB is the other range-partitioned engine I checked. It splits scans into per-node key spans (PartitionSpans), but its optimizer has no range-layout property (physical.Required). Its key-span range router is the closest match to this PR, including the error for rows outside every span. Bulk jobs such as restore use it, not SQL query plans.

So neither system puts a worker's key bounds in a partitioning property that the optimizer compares. DataFusion already goes further than Spark, because RangePartitioning carries split points and drives co-partitioning. Each DFD task also runs its own DataFusion plan, so its partitioning can describe only its own slice. Bounds fill that gap with key values. Empty partitions fill it by giving each task the stage-wide layout that Spark has. Bounds are Range-only, and every rule that compares range layouts (equality, can_interleave, #25483) has to define what they mean. Empty partitions need no new API but cost the partitions above. Could the PR description lay out that trade-off, with the measurement, so we can decide on it?

Because this PR is a bit large already, I can file a follow up issue for this.

A follow-up works for interleaving. As I read it, that's optimizer behavior behind the existing can_interleave signature. Could you file the issue and link it here?

The other half of my question decides what the public API means, so I don't think it can wait for a follow-up: how bounds take part when layouts are compared. Today has_same_layout ignores bounds and == includes them. #25483 will build its relationships on one of those, and its SourceFiner case needs bounds across tasks. For example, the last partition of Range(2, [10], end=20) maps into the first partition of Range(2, [20]) only because of end=20. If #25483 needs bounds to mean something else, we'd be changing the documented semantics of a released type. The API health policy asks us to avoid breaking changes and to deprecate instead, but a change to what == means can't be deprecated, so downstream code would change behavior without warning. Could we settle in this PR how bounds take part in layout comparison and document it on has_same_layout and PartialEq?

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jayshrivastava for the quick turnaround and for the detailed reply on the design. The adapt fix and the new doctest look good. Bounds become a permanent part of the public RangePartitioning once released, so I'd like us to finish comparing the alternatives before we adopt them.

Comment thread datafusion/common/src/partitioning.rs Resolved
.chain(end_bound.iter())
.cloned()
.collect::<Vec<_>>();
validate_range_split_points(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require each bound to lie strictly outside the retained samples

Ballista needs this one non-strict too. A task's bounds are the cuts on either side of its slice, so with repeated cuts start_bound can equal its first split point, and a task whose only partition sits between two equal cuts has start_bound == end_bound.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept this non strict for now.

@avantgardnerio avantgardnerio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As currently written, I think this PR is good from a Ballista range partitioning perspective - as long as the looser invariants remain for split points and bounds, we can adopt this for at least part of our plan.

@jayshrivastava

Copy link
Copy Markdown
Contributor Author

I pulled the equal split points work in to #27384. Once that's merged, I'll give this one a rebase.

Surajsuthar pushed a commit to Surajsuthar/datafusion that referenced this pull request Nov 17, 2026
## Which issue does this PR close?

Part of apache#26509. Extracted from apache#26593 following the [request to land
equal split points
separately](apache#26593 (review))
and the [request for standalone routing
tests](apache#26593 (review)).

## Rationale for this change

Repeated quantile cuts on skewed keys and intentionally empty partitions
can require equal adjacent split points. These layouts currently fail
validation. Allowing non-decreasing cuts preserves the requested
partition count: splits `[10, 10, 20]` route keys `[5, 10, 15, 25]` to
`[5]`, `[]`, `[10, 15]`, and `[25]`.

## What changes are included in this PR?

- Allow equality in `validate_range_split_points`, while continuing to
reject decreasing cuts and invalid tuple widths.
- Document that equal keys route after the last equal split point and
that intervening partitions are empty.
- Update constructor, file-scan, and range-transform validation tests,
and add `RepartitionExec` routing regressions for primitive and
two-column keys.

## What is the testing strategy for this PR?

Both new routing regressions fail with the old validation. The primitive
case exercises the specialized router; the compound case also checks
keys that differ in the second column around repeated boundaries.

- 5,978 unit tests across the five affected crates passed (two existing
ignored tests).
- 2,486 core and CLI tests passed, and all 549 SQL logic test files
passed.
- All checks in `dev/rust_lint.sh` passed, including
all-targets/all-features Clippy with `-D warnings`, generated
documentation checks, and Rust/HTML documentation builds. The final
checks were resumed after rebuilding a locally incompatible
`cargo-machete` binary.
- Eight `range_repartition` Criterion cases against `main`
(`b5bcea88a4`, `release-nonlto`) detected no regressions: uniform and
sequential keys, partitioner construction, and unchanged `RangeExpr`
routing at 8 and 512 partitions (30 samples per case, pinned to the same
CPU).

## Are there any user-facing changes?

`datafusion_common::validate_range_split_points` and the logical and
physical `RangePartitioning` constructors now accept equal adjacent
split points. Callers that relied on duplicate rejection must check for
equality explicitly. The DataFusion 57 upgrade guide documents this
behavior change.
@jayshrivastava
jayshrivastava force-pushed the js/range-partition-bounds branch from e0947f9 to 7048df1 Compare November 18, 2026 13:59
@github-actions github-actions Bot removed logical-expr Logical plan and expressions common Related to common crate labels Nov 18, 2026
@jayshrivastava

Copy link
Copy Markdown
Contributor Author

@stuhood @jayzhan211 #27384 has merged and I've rebased this PR

@jayzhan211 jayzhan211 added the api change Changes the API exposed to users of the crate label Nov 18, 2026
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jayshrivastava for splitting out #27384 and for the enforce-distribution tests, which made the bounds contract easy to check against the code.

Could you reply to @stuhood's review (#26593 (review)) on whether per-partition Statistics should carry this information instead of RangePartitioning? That review still requests changes, and #27374 is moving samples in that direction. I'd like that question answered on this PR before we merge, since bounds can't be moved out of RangePartitioning once they're released.

@mbutrovich

Copy link
Copy Markdown
Contributor

Since bounds are permanent once they ship, could we measure that cost before settling on them?

@avantgardnerio's reply in #26593 (comment) changes what bounds are for. They put a task's key range in the plan where a rule can use it, and the empty-partition alternative would need a separate API to say the same thing. With that argument, I no longer need the measurement.

Could you add a few sentences to the PR description on why bounds instead of empty partitions? The description becomes the commit message, and that's where someone will look when #25483 or #27106 has to define how bounds compose.

@mbutrovich

Copy link
Copy Markdown
Contributor
  1. Leave this as is, defer to the operator to handle bounds. If an operator declares Partitioning::Range with bounds and its input doesn't have bounds, the operator should handle this.

Option 1 works for me. It matches how sources declare partitioning and sortedness today, and the joins already take bounds only from the side whose rows they emit. I left a suggestion on the struct docs so authors of custom operators see the rule.

I have reverted the previous change were we strip the bounds.

At b25c830, RangePartitioning::adapt still sets both bounds to None, and I'd keep it that way.

Comment on lines -168 to +171
/// partition described by the split points. DataFusion will not validate this is
/// upheld.
/// partition described by the split points and bounds. DataFusion will not
/// validate that a source upholds this contract.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence covers sources, but your TestProjectionExec case (enforce_distribution.rs) shows that an operator overriding its output partitioning carries the same responsibility, and that nothing downstream catches a mistake. EnforceDistribution compares range layouts with has_same_layout, which ignores bounds. Could we say both here, so an operator author learns it from the docs instead of from wrong results?

Suggested change
/// partition described by the split points. DataFusion will not validate this is
/// upheld.
/// partition described by the split points and bounds. DataFusion will not
/// validate that a source upholds this contract.
/// Like other user-specified data properties such as sortedness, a plan that
/// declares range partitioning is responsible for placing each row in the
/// partition described by the split points and bounds. This applies to sources
/// and to operators that override their output partitioning. DataFusion will not
/// validate that a plan upholds this contract. `EnforceDistribution` compares
/// layouts with [`Self::has_same_layout`], which ignores bounds, so it never
/// inserts a filter or repartition because of them.

@stuhood

stuhood commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Sorry for the long delay here! Thanks @jayshrivastava for splitting out #27384, and thanks @mbutrovich for looping back on this.

I finally got around to filing #27600 to lay out the Statistics-based alternative in one place, along with a quick prototype and benchmark: https://github.com/paradedb/datafusion/blob/732c2171387105f567d66a27ccc573ddc2cbe2c8/BENCHMARK_EMPTY_PARTITIONS.md

Two quick notes on the discussion above:

  • On @avantgardnerio's point in feat: store start/end bounds in Partitioning::Range #26593 (comment) (that emptiness would stay private to the source or need a new API): with perf: share one StatisticsContext across all physical optimizer rules #26094's StatisticsContext, num_rows == Precision::Exact(0) is already visible to rules at any node in the plan, so rules can see which ranges a task owns directly from split_points + Exact(0) (including non-contiguous slices) without adding bounds to RangePartitioning.
    • It also "just works" across Range/Hash/RoundRobin.
  • On the Range(1600) runtime cost: with a small batch API on StatisticsContext (~0.1 ms across 1,600 partitions) and skipping Exact(0) inputs in RepartitionExec / CoalescePartitionsExec, zero empty leaf streams get executed and local coalesce/repartition run within noise of a 16-partition plan (0.56 ms vs 0.70 ms coalesce, 1.36 ms vs 1.47 ms Hash(16) repartition, and 10.6 ms vs 833 ms for a 1,600-way shuffle).

If folks are good with the #27600 direction, I'd be happy to clean up #27374 (it's still a draft, and will now need deprecation handling since 56.0.0 went out) to get it ready for review, and follow up with PRs for the batch stats and empty-partition skipping pieces as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate core Core DataFusion crate documentation Improvements or additions to documentation ffi Changes to the ffi crate physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

partitioning: store start and end bounds in Partitioning::Range

8 participants