Skip to content

[DO NOT MERGE] Spike managing saga timeouts - #6025

Draft
SzymonPobiega wants to merge 1 commit into
masterfrom
timeout-management-spike
Draft

SzymonPobiega wants to merge 1 commit into
masterfrom
timeout-management-spike

Conversation

@SzymonPobiega

@SzymonPobiega SzymonPobiega commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

The primary instance keeps a projection of saga timeouts reported by endpoints and lets an operator reschedule, trigger, or cancel a pending timeout. This spike implements RavenDB persistence only.

The design is in docs/saga-timeout-management-design.md, added by this pull request. The endpoint side is Particular/NServiceBus#7996, and the ServicePulse pages are Particular/ServicePulse#3166.

The design rests on one rule: the endpoint owns the timeout, and ServiceControl keeps a projection of it. ServiceControl never changes a timeout directly. An operator action is a request sent to the endpoint, which applies it to the saga and reports the result like any other change.

Changes

  • Local copies of the six report messages in namespace NServiceBus, so NServiceBus.EnclosedMessageTypes resolves by full name, as for EndpointUsageReport. One handler applies them through ISagaTimeoutDataStore.
  • Reports arrive out of order and more than once, so a timeout's state only moves forward, ordered by (Version, Rank). Requested and rescheduled have rank 0; triggered and canceled have rank 1, so a cancel wins a tie with a reschedule at the same version. Each history entry has a deduplication key, so a duplicate report changes nothing. A rejection for an unknown timeout creates an Unknown record that lists exclude.
  • SagaCompleted deletes the saga's timeouts and stores a completed-saga marker, so late reports for that saga are ignored.
  • RavenDB stores one SagaTimeouts/{TimeoutId} document with the current state and up to 100 history entries. Payloads are attachments named payload-v{Version}, and SagaTimeoutsIndex serves the list queries. Finished timeouts and markers expire after ErrorRetentionPeriod.
  • New routes:
    • GET api/timeouts with status, filters, paging, and sorting
    • GET api/timeouts/counts
    • GET api/timeouts/{id}
    • GET api/sagas/{sagaId}/timeouts
    • POST api/timeouts/{id}/reschedule, /trigger, and /cancel
    • DELETE api/timeouts/{id}
  • New permissions: error:timeouts:view, in the Read role, and error:timeouts:manage, in the Operate role.
  • An action checks expected_version and records a pending operation. It returns 202, 409 for a stale version or another pending operation, or 422 when the timeout is not pending or has no management address. The request then goes to the endpoint's satellite queue from a handler, in the handler's transport transaction, and is written to the message action audit log.
  • A timeout is overdue when it is still pending 5 minutes after its due time. This is computed at query time.

Constraints and deviations from the design

  • The SQL Server and PostgreSQL persisters register a store that throws NotSupportedException. A report on a SQL-backed instance fails and goes to the error queue. Their test projects exclude the new shared persistence tests.
  • Not implemented: grouping, the change history list, failed-trigger tracking from error ingestion, event log entries, domain and integration events, the saga diagram additions, and the SQL retention sweep.
  • If a report and SagaCompleted for the same saga are processed at the same time, a timeout can be left behind. The completion finds the documents to delete through the index. Forget removes such a timeout.
  • The design document's status line still says ServiceControl implements none of this.

Verification

  • New persistence tests on RavenDB cover apply ordering, duplicates, completion and reports after it, rejections creating Unknown timeouts, operator request outcomes, and forget. ServiceControl.Persistence.Tests.RavenDB: 371 passed.
  • ServiceControl.UnitTests: 461 passed, including satellite request headers and deserializing a report the way the endpoint sends it. ServiceControl.Infrastructure.Tests: 115 passed.
  • A new RavenDB acceptance test sends a raw NServiceBus.SagaTimeoutRequested with no assembly name and no message id header, then reads it through the API.
  • Not run: the SQL Server and PostgreSQL test projects and the full acceptance suites. The SQL projects compile.
  • Ran end to end with RabbitMQ, an endpoint built from NServiceBus#7996, and ServicePulse#3166.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants