Skip to content

fix(fault_manager): let report_fault hold a burst of requests - #789

Open
bburda wants to merge 1 commit into
mainfrom
fix/fault-reporter-humble-delivery
Open

bburda wants to merge 1 commit into
mainfrom
fix/fault-reporter-humble-delivery

Conversation

@bburda

@bburda bburda commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

~/report_fault used the default service QoS, so its request queue held 10 requests. When more reporters reported at once than that, the middleware dropped the extra requests without an answer, and under load some reports never reached the manager. The queue now holds 100 requests.

KEEP_ALL is not used: the response writer shares this QoS, and a client that stops acknowledging would fill its history until sending a response fails.

The FaultReporter docs now also say that a report answered later than ack_timeout_sec is sent again and processed once per copy.


Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

test_delivery in ros2_medkit_fault_reporter covers it (test_burst_of_fresh_reporters_loses_nothing, test_events_of_one_code_arrive_in_order). It fails on main on Humble with Fast DDS on a 4-core machine and passes with this change.

The Pixi (humble) workflow is the CI job that runs this combination. It runs on push to main and on schedule, so it was started by hand on this branch.


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

~/report_fault used the default service QoS, so its request queue held
10 requests. When more reporters reported at once than the queue held,
the middleware dropped requests without an answer. Under load the
FaultReporter retries met a full queue too, and some reports used all
their attempts before they reached the manager.

The queue now holds 100 requests. With a deeper queue a late answer is
more likely than a lost request, so the FaultReporter docs now say that
a report answered too late is sent again and every copy is processed.
@bburda
bburda requested a review from mfaferek93 November 15, 2026 21:01
@bburda bburda self-assigned this Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] fault_manager drops fault reports when many reporters report at once

1 participant