Skip to content

Fix orchestration completion/termination race condition - #143

Merged
Simon Porter (simonporter) merged 1 commit into
masterfrom
terminate-race
Mar 3, 2018
Merged

Simon Porter (simonporter) merged 1 commit into
masterfrom
terminate-race

Conversation

@cgillum

Copy link
Copy Markdown
Member

This fix addresses an issue brought up by Durable Functions customers: Azure/azure-functions-durable-extension#146.

The issue happens when DTFx receives a terminate message in the same message batch that would normally complete the orchestration. The terminate action gets executed, but then the completion action also gets executed, resulting in a state corruption error:

TaskOrchestrationDispatcher-f79977801ceb41aa82701d473427e52b-0: Exception while processing workItem 87e126f0d4f54d9ca092bf67c10cf4e0 Exception: System.InvalidOperationException : Multiple ExecutionCompletedEvent found, potential corruption in state storage at DurableTask.Core.OrchestrationRuntimeState.SetMarkerEvents(HistoryEvent historyEvent)
 at DurableTask.Core.OrchestrationRuntimeState.AddEvent(HistoryEvent historyEvent, Boolean isNewEvent)
 at DurableTask.Core.OrchestrationRuntimeState.AddEvent(HistoryEvent historyEvent)
 at DurableTask.Core.TaskOrchestrationDispatcher.ProcessWorkflowCompletedTaskDecision(OrchestrationCompleteOrchestratorAction completeOrchestratorAction, OrchestrationRuntimeState runtimeState, Boolean includeDetails, Boolean& continuedAsNew)
 at DurableTask.Core.TaskOrchestrationDispatcher.<OnProcessWorkItemAsync>d__17.MoveNext() --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at DurableTask.Core.WorkItemDispatcher`1.<ProcessWorkItemAsync>d__34.MoveNext()

This is very similar to #25 (which I think is actually fixed, based on my reading of the code) but is just a slightly different variation that involves orchestration completion.

@simonporter
Simon Porter (simonporter) merged commit 1b3efef into master Mar 3, 2018
@cgillum
Chris Gillum (cgillum) deleted the terminate-race branch March 11, 2018 00:08
zheg pushed a commit to zheg/durabletask that referenced this pull request Mar 27, 2018
@jeffputz

Copy link
Copy Markdown

Are you guys planning a release soon? I noticed the last one is a year+ old.

@cgillum

Copy link
Copy Markdown
Member Author

There was a release 3 days ago which includes this fix: https://www.nuget.org/packages/DurableTask.Core/2.0.0.4-preview5

@jeffputz

Copy link
Copy Markdown

So there is. Apologies, I don't always math good. :)

Simon Porter (simonporter) pushed a commit that referenced this pull request May 18, 2018
* Fix for the issue where issuing TerminateInstanceAsync twice from client before the orchestration is picked up by dispatcher leads to a blocked orchestration (the processing keeps abandoning the orchestration). (#113)

Only the first terminate request is honored and subsequent terminates are ignored. (That makes the fix simple and hopefully not a problematic behavior from user's perspective)

* Handle the case where an orchestration completes and terminates at the same time (#143)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants