You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Invalid plans result in SortPreservingRepartitionExec with no SortExprs #7794
While integrating #7671 and turning on with_bounded_order_preserving_variants (since renamed to prefer_existing_sort) in IOx, some of our plans end up with a SortPreservingRepartitionExec that had no sort expressions
This caused a very non intuitive error much later in the plan -- interleave paniced with Panic: range end index 8192 out of range for slice of length 1
| problem --> SortPreservingRepartitionExec: partitioning=Hash([iox::measurement@0], 16), input_partitions=16 <-- this has no sport expressions
| RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=2
| UnionExec
| ProjectionExec: expr=[cpu as iox::measurement, 0 as time, AVG(cpu.usage_idle)@0 as mean, NULL as mean_1]
| AggregateExec: mode=Final, gby=[], aggr=[AVG(cpu.usage_idle)]
| CoalescePartitionsExec
|
I am still working out a smaller reproducer for this issue, but filing the ticket for more visibility.
Thanks @alamb for great explanation and locating the issue. I have open the #7811, which I believe solves this issue. As far as I can tell, there are mainly two issues here.
First one is EnforceSorting moves up the SortExec that satisfy an executer right under the executor. Then during pushdown pass that SortExec is tried to move down, if it is helpful (In your case, SortExec is not pushed down through RepartitioinExec).
Second one is when relative position of the SortExec changes, some of the executors in between may be invalid. Which is the case, in this issue.
The PR solves the second problem, while removing SortExec it replaces executors on top of it with variants that do not preserve order.
However, after this PR EnforceSorting may still move up some SortExec to the executor that require it.
I tried to replicate the issue in the PR, however I did not use the exact same plan here. Hence this explanation is educated guess. If #7811 doesn't resolve this issue, please let me know.
Activity
alamb commentedon Oct 11, 2023
I got a few more plans, with #7796 added. This time I can see that the plan lost the expressions after the
EnforceSortingpassAfter the sort preserving repartition exec the sort exprs are still here:
But after "EnforceSorting" they are gone (why is there a sort above them???)
mustafasrepo commentedon Oct 12, 2023
Thanks @alamb for great explanation and locating the issue. I have open the #7811, which I believe solves this issue. As far as I can tell, there are mainly two issues here.
EnforceSortingmoves up theSortExecthat satisfy an executer right under the executor. Then duringpushdownpass thatSortExecis tried to move down, if it is helpful (In your case,SortExecis not pushed down throughRepartitioinExec).SortExecchanges, some of the executors in between may be invalid. Which is the case, in this issue.The PR solves the second problem, while removing
SortExecit replaces executors on top of it with variants that do not preserve order.However, after this PR
EnforceSortingmay still move up someSortExecto the executor that require it.I tried to replicate the issue in the PR, however I did not use the exact same plan here. Hence this explanation is educated guess. If #7811 doesn't resolve this issue, please let me know.
alamb commentedon Oct 12, 2023
Thank you so much @mustafasrepo . I will test locally with #7811 and report back
alamb commentedon Oct 12, 2023
Update is that #7811 fixed the problem I was seeing. Thank you 🙏