You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have observed that sometimes an execution plan with a small input never finishes (the future returned by the ExecPlan::finished() method is never marked as finished), even though the generator in the sink node is exhausted and has returned nullopt.
This issue seems to happen at random, the same plan with the same input sometimes works (the plan is marked finished) and sometimes it doesn't. Since the ExecPlanImpl destructor forces the executing thread to wait for the plan to finish (when the plan has not yet finished) we enter in a deadlock waiting for a plan that never finishes.
Since this has only happened with small inputs and not in a deterministic way, we believe the issue might be in the ExecPlan::StartProducing method.
Our hypothesis is that after the plan starts producing on each node, each node schedules their tasks and they are immediately finished (due to the small input) and somehow the callback that marks the future finished_ finished is never executed.
Weston Pace / @westonpace:
Are you able to provide some more information on the structure of the plan? Can you print the plan? What is your input? Are you reading from files or in-memory tables or some kind of record batch reader?
My guess would be some kind of deadlock in mutexes from callbacks firing more quickly than we'd expect. It should be fixable if we can find a way to reproduce semi-reliably.
The SourceNode is using a readerGenerator from a RecordBatchReader. This last reader comes from a dataset scanner (the dataset just contains a single parquet file).
We are using the dataset scanner instead of the scan node because in earlier versions we saw it was faster than using the scan node. I believe this has changed and there's been improvements in recent versions and maybe we should just use the scan node again.
As a matter of fact, analysing the OpenTelemetry traces I've seen another plan printed:
Could it be that the dataset scanner implementation is in fact another Acero plan ? (I have no idea of the implementation details sorry). If so, we could just use a single plan with the scan node and forget the dataset scanner.
As for the parquet file we are using, the metadata header is as follows:
As you can see it's only 470 rows, so the execution plan finishes very quickly.
I hope this information helps. From our side we'll replace the dataset scanner and record batch reader shenanigans with the scan node to see if the deadlock does not occur or at least have a cleaner code.
The program is just an infinite loop that makes and executes the same simple plan with just a single row of input. Usually after a few iterations the program enters in a deadlock waiting for the plan to finish.
Closing: this looks like the group-by node deadlock fixed in #33700 (released in 11.0.0, see also #15243), which landed shortly after this report, and the reprex no longer hangs on current Arrow. If you can still reproduce it on a recent release, please open a new issue with the updated reprex.
Activity
asfimport commentedon Dec 9, 2022
Weston Pace / @westonpace:
Are you able to provide some more information on the structure of the plan? Can you print the plan? What is your input? Are you reading from files or in-memory tables or some kind of record batch reader?
My guess would be some kind of deadlock in mutexes from callbacks firing more quickly than we'd expect. It should be fixable if we can find a way to reproduce semi-reliably.
asfimport commentedon Dec 10, 2022
Pau Garcia Rodriguez:
Hello, thank you for your quick answer.
The explicit plan we use is as follows:
The SourceNode is using a readerGenerator from a RecordBatchReader. This last reader comes from a dataset scanner (the dataset just contains a single parquet file).
We are using the dataset scanner instead of the scan node because in earlier versions we saw it was faster than using the scan node. I believe this has changed and there's been improvements in recent versions and maybe we should just use the scan node again.
As a matter of fact, analysing the OpenTelemetry traces I've seen another plan printed:
Could it be that the dataset scanner implementation is in fact another Acero plan ? (I have no idea of the implementation details sorry). If so, we could just use a single plan with the scan node and forget the dataset scanner.
As for the parquet file we are using, the metadata header is as follows:
As you can see it's only 470 rows, so the execution plan finishes very quickly.
I hope this information helps. From our side we'll replace the dataset scanner and record batch reader shenanigans with the scan node to see if the deadlock does not occur or at least have a cleaner code.
Best regards,
Pau.
asfimport commentedon Dec 11, 2022
Pau Garcia Rodriguez:
I managed to reproduce it with this simple program:
The program is just an infinite loop that makes and executes the same simple plan with just a single row of input. Usually after a few iterations the program enters in a deadlock waiting for the plan to finish.
I hope this helps to find the root cause.
thisisnic commentedon Nov 9, 2026
Closing: this looks like the group-by node deadlock fixed in #33700 (released in 11.0.0, see also #15243), which landed shortly after this report, and the reprex no longer hangs on current Arrow. If you can still reproduce it on a recent release, please open a new issue with the updated reprex.