Repository navigation
feat: store start/end bounds in Partitioning::Range - #26593
jayshrivastava wants to merge 12 commits into
Conversation
Partitioning::Range
|
👋🏽 @stuhood @saadtajwar @avantgardnerio would any of you be interested in reviewing this? This is related to #22395, which you all have participated in. Please lmk! |
Hey @jayshrivastava - thanks for tagging, happy to review this! I'll try to get to this in the next couple of days :) |
|
Thanks @saadtajwar! |
| /// according to the provided sort options. Equal split points imply empty partitions. | ||
| pub fn validate_range_split_points( | ||
| split_points: &[SplitPoint], | ||
| sort_options: &[SortOptions], | ||
| ) -> Result<()> { | ||
| let width = sort_options.len(); | ||
| for (idx, split_point) in split_points.iter().enumerate() { | ||
| let split_point_width = split_point.values().len(); | ||
| if split_point_width != width { | ||
| return _plan_err!( | ||
| "Range partitioning split point {idx} has width {split_point_width}, but ordering has width {width}" | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| for (idx, split_points) in split_points.windows(2).enumerate() { | ||
| if compare_rows( | ||
| split_points[0].values(), | ||
| split_points[1].values(), | ||
| sort_options, | ||
| )? != Ordering::Less | ||
| )? == Ordering::Greater |
There was a problem hiding this comment.
Hmm out of curiosity, can I ask if there's context around the addition of equal split points? Is there a case for distributed execution in which it would be helpful to have an empty partition that I may be missing?
Another thing which might be useful is to be able to pad empty partitions. Ex. use duplicate split points such as 10, 10, 10, 10 to indicate empty partitions.
I see the above comment in #26509, but would definitely like to understand use cases for the empty partitions :)
There was a problem hiding this comment.
The main reason is that padding empty partitions is supported Partitioning::Unknown. For example, I I can add an empty file group. Doing the same when a data source is Partitioning::Range introduces an error.
In distributed-datafusion, this feature ensure all tasks (horizontally distributed parts of the plan) have the same number of partitions. This simplifies the code a lot. I imagine the same is true for ballista etc but I'm not 100% sure. We could move away from this if anyone feels strongly, but I figured it's a pretty inoffensive change to make in this repo.
There was a problem hiding this comment.
I see, makes sense to me! Thanks!
There was a problem hiding this comment.
In distributed-datafusion, this feature ensure all tasks (horizontally distributed parts of the plan) have the same number of partitions.
We could move away from this if anyone feels strongly, but I figured it's a pretty inoffensive change to make in this repo.
I'd rather DataFusion not take on that invariant without more discussion. As far as I can find, it doesn't have one today. Partition counts already differ from operator to operator, and the only places that need equal counts are children that must be co-partitioned (compatible_co_partitioning_layout) and interleaved inputs (can_interleave). I'd like to keep DataFusion open to running operators with different partition counts, for example an operator that doesn't need as many partitions coalescing adjacent ranges (the SourceFiner case in #25483). Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ.
Bounds already let a task have fewer partitions without padding. In Example 4 of datafusion-contrib/datafusion-distributed#782, the last task could be Range(2, [60], start=50) instead of split points [50, 60] with start=50 and an empty first partition. Could we keep validate_range_split_points strict, require each bound to lie strictly outside the retained samples, and leave padding to DFD? That also removes the empty range where start_bound == end_bound. Once DataFusion accepts equal split points, tightening the check later would reject layouts that downstream code already builds. If the check stays strict, the upgrade-guide entry I asked for on line 69 only needs to cover the bounds fields.
There was a problem hiding this comment.
Making empty partitions irrepresentable in the Partitioning::Range specification can be cornering.
For example, having an empty FileGroup in a parquet scan is something that can happen, there's no invariant that prevent that, however, that'd automatically make it incompatible with the current specification of Partitioning::Range. At some point, FileScanConfig might want to expose range partitioning here https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/file_scan_config/mod.rs#L908-L940, so how should it react in the event of one of the FileGroups being empty?
One other idea that comes to mind is to explicitly model empty partitions somehow, maybe making the split points accept a ScalarValue::Null and accounting for it?
Regardless of the mechanism, I do see value in making empty partitions representable.
There was a problem hiding this comment.
Partition counts already differ from operator to operator, and the only places that need equal counts are children that must be co-partitioned (compatible_co_partitioning_layout) and interleaved inputs (can_interleave). I'd like to keep DataFusion open to running operators with different partition counts, for example an operator that doesn't need as many partitions coalescing adjacent ranges (the SourceFiner case in #25483). Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ.
I would expect can_interleave and compatible_co_partitioning_layout to be improved so that we are able to interleave arbitrary partitioned data. Ex. Range(3, [60, 80], start=50) should be interleaved with Range(4, [60, 80]). You mentioned this below as well. I'll file an issue.
Also, my experience has been that, if the user wants to avoid a RepartitionExec or wants an InterleaveExec, they have to make sure the correct partitioning properties are declared on their data source. I've even gone as far to declare data sources as Hash() partitioned when they are not. This is to say that I think a user would probably go out of their way to declare interleavable and/or compatible, not necessarily relying on datafusion.
Padding with equal split points pushes the other way: it raises partition counts to match instead of letting them differ
I'd say the uses for padding are user-defined, like our invariant in DFD and the KLL example Brent mentioned above.
Padding would not really be used to achieve co-partitioning. For example, I don't see a way to pad Range(2, [50]) to make it compatible with Range(3, [50, 60]). Padding with Range(3, [50, 50]) doesn't work. Converting to Range(3, [50, 60]) requires a repartition, splitting up the last partition.
In Example 4 of datafusion-contrib/datafusion-distributed#782, the last task could be Range(2, [60], start=50) instead of split points [50, 60] with start=50 and an empty first partition.
The main distinction is that the former has 2 partitions and latter has 3. You force the user to have an additional tasks spawned if they want to add bounds to Range partitioning.
Padding empty partitions also introduces additional tasks, but the user can opt out of this by simply not padding.
There was a problem hiding this comment.
🤔 I'd also double check how easy it'd be to remove the invariant in DFD about needing the same partition count in all task-specialized plans in a stage.
If we were able to remove that invariant, will it avoid introducing some sense of padding in this project?
There was a problem hiding this comment.
Just wanted to say thank you for bringing some more eyes and perspectives to this topic!
There was a problem hiding this comment.
And thank you for being flexible in your understanding! Always a pleasure...
There was a problem hiding this comment.
+1. Appreciate the details reviews from everyone :)
|
FAQ
Some types don't have a max or a reasonable value ex. what's the max of a string? "zzzzzz"? For variable length types too
Tests around range satisfaction. Generally, a join may join two different ranges, with or without bounds. 50% of this PR is just adding test coverage around that. For example, the join type influences the behavior here. Happy to delete tests if people thing any are less useful. |
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @jayshrivastava! The join-type test makes it easy to see which side's bounds each join can inherit.
Should we settle the alternative @stuhood raised in datafusion-contrib/datafusion-distributed#628 (comment) before bounds become part of the public range contract? There, every task keeps the full Range(N, split_points) and the partitions outside its slice are empty, which needs no new fields on RangePartitioning. Spark describes the same kind of slice by index: a coalesced AQE read gives each reader a span of reducer partitions and keeps the original RangePartitioning with a smaller count (AQEShuffleReadExec). As I read your reply in datafusion-contrib/datafusion-distributed#628 (comment) and the padding thread here, this PR still pads tasks with empty partitions, so what the bounds save is the N - N/W idle partitions on each task. In exchange, every range rule has to account for bounds. For example, equality now includes them, so can_interleave rejects inputs that differ only in bounds, and the layout relationships in #25483 will need to define how bounds compose. Is the idle-partition overhead the remaining cost of that approach, or is there a plan shape it can't express?
Stu's approach is valid. I also have a similar one in datafusion-contrib/datafusion-distributed#782. I decided to publish this PR because I felt that those two approaches diverge from our goal of keep DFD as close to DF as possible and as thin of a layer over the single node implementation. Introducing these bounds reduces the complexity of datafusion distributed very significantly, and the lines of code as well. Ballista opted to implement range shuffles without introducing bounds to DF and you can see the tradeoffs that needed to be made. Consider that we want to distributed this across two tasks In ballista, the The Your suggestion is to have each task advertise all partitions but mark some as empty like so That is a reasonable alternative, but those empty partitions have an extra cost, the
On Spark, I think the distinction is that its physical
Regarding idle partitions, the user has full control here. They can choose to not have them. This PR only introduces the option to pad. On However, we may want to interleave them and use this layout. Because this PR is a bit large already, I can file a follow up issue for this. Because bounds are new, I doubt many users will run into this. When someone does, that optimization can be implemented fairly easily. |
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @jayshrivastava
This looks good, just couple of items.
Thanks, that tells me the difference is run-time cost and DFD code, not a plan the alternative can't express. Since bounds are permanent once they ship, could we measure that cost before settling on them? Each task pays for
As I read it, the same index argument works in DataFusion under the empty-partition alternative. Every task advertises the same I think the more useful comparison with Spark is where it keeps the slice. Its Spark's coalescing also covers a case DFD will hit: partitions too small to be worth a worker each. AQE packs contiguous reducer partitions up to a target size ( CockroachDB is the other range-partitioned engine I checked. It splits scans into per-node key spans ( So neither system puts a worker's key bounds in a partitioning property that the optimizer compares. DataFusion already goes further than Spark, because
A follow-up works for interleaving. As I read it, that's optimizer behavior behind the existing The other half of my question decides what the public API means, so I don't think it can wait for a follow-up: how bounds take part when layouts are compared. Today |
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @jayshrivastava for the quick turnaround and for the detailed reply on the design. The adapt fix and the new doctest look good. Bounds become a permanent part of the public RangePartitioning once released, so I'd like us to finish comparing the alternatives before we adopt them.
| .chain(end_bound.iter()) | ||
| .cloned() | ||
| .collect::<Vec<_>>(); | ||
| validate_range_split_points( |
There was a problem hiding this comment.
require each bound to lie strictly outside the retained samples
Ballista needs this one non-strict too. A task's bounds are the cuts on either side of its slice, so with repeated cuts start_bound can equal its first split point, and a task whose only partition sits between two equal cuts has start_bound == end_bound.
There was a problem hiding this comment.
Kept this non strict for now.
avantgardnerio
left a comment
There was a problem hiding this comment.
As currently written, I think this PR is good from a Ballista range partitioning perspective - as long as the looser invariants remain for split points and bounds, we can adopt this for at least part of our plan.
|
I pulled the equal split points work in to #27384. Once that's merged, I'll give this one a rebase. |
## Which issue does this PR close? Part of apache#26509. Extracted from apache#26593 following the [request to land equal split points separately](apache#26593 (review)) and the [request for standalone routing tests](apache#26593 (review)). ## Rationale for this change Repeated quantile cuts on skewed keys and intentionally empty partitions can require equal adjacent split points. These layouts currently fail validation. Allowing non-decreasing cuts preserves the requested partition count: splits `[10, 10, 20]` route keys `[5, 10, 15, 25]` to `[5]`, `[]`, `[10, 15]`, and `[25]`. ## What changes are included in this PR? - Allow equality in `validate_range_split_points`, while continuing to reject decreasing cuts and invalid tuple widths. - Document that equal keys route after the last equal split point and that intervening partitions are empty. - Update constructor, file-scan, and range-transform validation tests, and add `RepartitionExec` routing regressions for primitive and two-column keys. ## What is the testing strategy for this PR? Both new routing regressions fail with the old validation. The primitive case exercises the specialized router; the compound case also checks keys that differ in the second column around repeated boundaries. - 5,978 unit tests across the five affected crates passed (two existing ignored tests). - 2,486 core and CLI tests passed, and all 549 SQL logic test files passed. - All checks in `dev/rust_lint.sh` passed, including all-targets/all-features Clippy with `-D warnings`, generated documentation checks, and Rust/HTML documentation builds. The final checks were resumed after rebuilding a locally incompatible `cargo-machete` binary. - Eight `range_repartition` Criterion cases against `main` (`b5bcea88a4`, `release-nonlto`) detected no regressions: uniform and sequential keys, partitioner construction, and unchanged `RangeExpr` routing at 8 and 512 partitions (30 samples per case, pinned to the same CPU). ## Are there any user-facing changes? `datafusion_common::validate_range_split_points` and the logical and physical `RangePartitioning` constructors now accept equal adjacent split points. Callers that relied on duplicate rejection must check for equality explicitly. The DataFusion 57 upgrade guide documents this behavior change.
e0947f9 to
7048df1
Compare
|
@stuhood @jayzhan211 #27384 has merged and I've rebased this PR |
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @jayshrivastava for splitting out #27384 and for the enforce-distribution tests, which made the bounds contract easy to check against the code.
Could you reply to @stuhood's review (#26593 (review)) on whether per-partition Statistics should carry this information instead of RangePartitioning? That review still requests changes, and #27374 is moving samples in that direction. I'd like that question answered on this PR before we merge, since bounds can't be moved out of RangePartitioning once they're released.
@avantgardnerio's reply in #26593 (comment) changes what bounds are for. They put a task's key range in the plan where a rule can use it, and the empty-partition alternative would need a separate API to say the same thing. With that argument, I no longer need the measurement. Could you add a few sentences to the PR description on why bounds instead of empty partitions? The description becomes the commit message, and that's where someone will look when #25483 or #27106 has to define how bounds compose. |
Option 1 works for me. It matches how sources declare partitioning and sortedness today, and the joins already take bounds only from the side whose rows they emit. I left a suggestion on the struct docs so authors of custom operators see the rule.
At b25c830, |
| /// partition described by the split points. DataFusion will not validate this is | ||
| /// upheld. | ||
| /// partition described by the split points and bounds. DataFusion will not | ||
| /// validate that a source upholds this contract. |
There was a problem hiding this comment.
This sentence covers sources, but your TestProjectionExec case (enforce_distribution.rs) shows that an operator overriding its output partitioning carries the same responsibility, and that nothing downstream catches a mistake. EnforceDistribution compares range layouts with has_same_layout, which ignores bounds. Could we say both here, so an operator author learns it from the docs instead of from wrong results?
| /// partition described by the split points. DataFusion will not validate this is | |
| /// upheld. | |
| /// partition described by the split points and bounds. DataFusion will not | |
| /// validate that a source upholds this contract. | |
| /// Like other user-specified data properties such as sortedness, a plan that | |
| /// declares range partitioning is responsible for placing each row in the | |
| /// partition described by the split points and bounds. This applies to sources | |
| /// and to operators that override their output partitioning. DataFusion will not | |
| /// validate that a plan upholds this contract. `EnforceDistribution` compares | |
| /// layouts with [`Self::has_same_layout`], which ignores bounds, so it never | |
| /// inserts a filter or repartition because of them. |
|
Sorry for the long delay here! Thanks @jayshrivastava for splitting out #27384, and thanks @mbutrovich for looping back on this. I finally got around to filing #27600 to lay out the Two quick notes on the discussion above:
If folks are good with the #27600 direction, I'd be happy to clean up #27374 (it's still a draft, and will now need deprecation handling since |
Important
Fork CI setup: For pull requests from a fork to
main, enable workflows in your fork's Actions tab before you push.See the fork CI setup guide.
Which issue does this PR close?
Rationale for this change
Most of the context is in the issue above.
The main use case is that I want to partition a query across multiple machines while assigning
Partitioning::Rangeto each machine. Because we don't store a start and end bound, you lose information:Because split point
20is lost, both machines have overlapping rangesx >= 10andx < 30. This means you cannot, for example, coalesce them back into the original partitioning. In that scenario, you you have to basically fall back toPartitioning::Unknown(4)This change adds optional start and end bounds, so you can split up the range like so:
Now, you preserve information.
Partitioning::Range(4)instead ofPartitioning::Unknown(4)What changes are included in this PR?
The diff is large but it's a very straightforward, mechanical change.
Partitioning::Rangenow has 2 new fields.Nonemeans "unbounded", which is equivalent to the default today. The difference is that now, as a user, I can split this up intoNranges without losing information by storing split points in the bounds.Minor change: split points no longer need to be strictly increasing. This allows a user to pad empty ranges if they would like.
To support bounds in
Partitioning::Range,RepartitionExechas some minor changes to the router.What is the testing strategy for this PR?
Unit tests and existing tests.
Are there any user-facing changes?
RangePartitioningcan now optionally store bounds in addition to internal split points.FFI_RangePartitioninghas two new fields. Because this changes the struct's C layout, this is technically an API change.