Skip to content

fault_manager: optional per-source fault records, chosen per fault code - #699

Draft
bburda wants to merge 40 commits into
mainfrom
feat/fault-identity-owner-key
Draft

bburda wants to merge 40 commits into
mainfrom
feat/fault-identity-owner-key

Conversation

@bburda

@bburda bburda commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

A fault code can now keep one record per reporting source. The choice is made per code:

  • By default a code keeps one record for every source, as before: one debounce counter, reporting_sources lists
    every source, and a clear or get without a source works.
  • per_source: true on the code's entry in fault_thresholds.yaml gives each source its own record, with its own
    status, debounce, severity, occurrences, freeze frame, snapshots, rosbag links and near misses. ClearFault,
    GetFault, GetSnapshots and GetRosbag take a trailing source_id; an empty one is refused as ambiguous: when
    several records share the code. A code whose mode changes keeps its stored records and logs a warning at startup.

Also in this PR:

  • Correlation matches across sources and keeps its relations per record. A root cause that is cleared or healed releases
    the symptoms it muted, also with skip_correlation_auto_clear; a re-reported root keeps them; show_as_single hides
    the members other than the current representative, as the tutorial describes. A record stays muted while another
    root cause or an active cluster still mutes it. These fix behaviour that main has for every code.
  • SQLite and PostgreSQL stores written by an earlier release migrate on open. The migration is one way: keep a copy of
    the store before upgrading.
  • The OPC UA plugin names the source it reported under in every clear. A bridge with a node map no longer treats
    records under host-derived ids as its own, so two bridges on one host do not clear each other's records.
  • MutedFaultInfo, ClusterInfo and GetRosbag gain owner fields. Every client of the fault manager has to be
    rebuilt against these messages.

Gateway support for per-source records is the next PR in this stack.


Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

colcon build --symlink-install --packages-up-to ros2_medkit_fault_manager ros2_medkit_opcua ros2_medkit_integration_tests
colcon test --packages-select ros2_medkit_fault_manager ros2_medkit_opcua --parallel-workers 1
colcon test --packages-select ros2_medkit_integration_tests --ctest-args -R "fault|correlation|rosbag"

src/ros2_medkit_fault_manager/test/integration/test_fault_identity.test.py shows both modes over the services: a
default code reported by three sources is one record, a per_source code is three. PostgreSQL: build with
-DPOSTGRES_SUPPORT=ON and set ROS2_MEDKIT_TEST_PG_CONN (the CI postgres job does both).


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

@bburda bburda self-assigned this Sep 26, 2026
for (const auto & prc : pending_root_causes_) {
// A root cause only explains its OWN reporter's faults. Without this the first
// owner to report a root-cause code would mute every other owner's symptom.
if (prc.fault_id.owner != id.owner) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This owner check (and the (rule_id, owner) cluster key at line 437) turns every hierarchical and cluster relation into a same-owner relation, which #698 does not ask for and which silently retires the documented cross-node behaviour: the tutorial's E-stop cascade and sensor storm span nodes, docs/tutorials/fault-correlation.rst still explains skip_correlation_auto_clear as the guard against cascading into "apps in other entities", and the fault_manager integration tests only pass because the symptom source was rewritten from /motor_node to /estop_node (test_integration.test.py:814-816) and the storm from /node{i + 1} to /node1. Keep matching by code across owners and make the relation per record (mute and cascade each symptom FaultId on its own, which is what "muting hides a record rather than a code" means), or if same-owner correlation is the intended design, call it out as a breaking change in the PR and update the tutorial.

Comment thread src/ros2_medkit_fault_manager/README.md Outdated
The rebuild is one way. A fault manager from an earlier release opened on a migrated store aborts
on the first report of a fault code the store does not hold yet
(`NOT NULL constraint failed: faults.owner`), and on a fault code two sources share it writes one
source's data into the other source's row. If a rollback may be needed, keep a copy of `faults.db`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Aborts on the first report of a fault code the store does not hold yet" understates the downgrade: main's initialize_schema runs DELETE FROM rosbag_files WHERE id NOT IN (SELECT MAX(id) FROM rosbag_files GROUP BY fault_code, file_path) on open, which silently drops one owner's link row for every bag two owners of a code share (the burst case finalize_post_fault_recording now produces), before any report and before the NOT NULL abort, so an earlier release opened on a migrated store loses rows quietly rather than failing loudly. Either guard the old code path (a faults view over a renamed table makes main's startup UPDATE faults SET last_occurred_ns ... throw before it writes anything) or state here that opening a migrated file with an earlier release deletes rows and is unsupported, with the backup as the only way back.

# Reporting source that owns the fault record (the source_id used in ReportFault).
# Together with fault_code it identifies one record. Empty: the call applies only
# when exactly one record carries fault_code, otherwise it fails as ambiguous.
string source_id

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The in-tree OPC UA plugin still calls this service without the new field: OpcuaPlugin::send_clear_fault (src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp:1284, reached from on_alarm_change, on_event_alarm and the FaultProvider clear_fault) sends only fault_code, while its reports carry entity_id as source_id. PLC_COMMS_LOST is raised per component, so once a second emitter of that code has ever reported to the same fault_manager the plugin's clear is refused as ambiguous (cleared records count too) and, since the plugin never reads the response, the fault stays CONFIRMED with no error on the plugin side. Pass the entity id through as request->source_id at the three call sites.

/// Pending cluster with steady_clock timestamp for window tracking
struct PendingCluster {
ClusterData data;
std::string owner; ///< Every member record of this cluster has this owner

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clusters are per owner now, but ClusterInfo.msg still carries bare fault_codes and no owner, so a consumer of ListFaults(include_clusters=true) can neither address the member records with the single-record services nor tell two clusters of one rule apart beyond the counter suffix in cluster_id. MutedFaultInfo got source_id for exactly this reason - give ClusterInfo the same field, filled from PendingCluster::owner.

auto fault = storage_->get_fault(reclassified_id);
if (fault) {
audit_transition(kTransitionCleared, *fault, "startup_reclassify", reclassified_at_ns);
audit_transition(kTransitionCleared, *fault, reclassified_id.owner, reclassified_at_ns);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing the owner into source for the automatic transitions (here, the timer at line 460, the heal at 905, the cascade at 1060) drops information the chain used to carry: main wrote startup_reclassify, auto_confirm_timer, auto_heal and clear_service, and now a cleared row from an operator's ClearFault, a startup reclassification and a correlation cascade are byte-identical, as are confirmed by report and by timer, which the README's new claim that "the transition column already says what moved the row" (README.md:253) does not cover. Keep the driver in the row alongside the owner, for example a new field folded into canonicalize only when non-empty so existing chains still verify, and correct the README sentence.

for (const auto & prc : pending_root_causes_) {
// A root cause only explains its OWN reporter's faults. Without this the first
// owner to report a root-cause code would mute every other owner's symptom.
if (prc.fault_id.owner != id.owner) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops cross-node correlation, which is the main use case. In the README estop_cascade example ESTOP_001 comes from the safety node and MOTOR_COMM_* from the motor controllers, so with this check nothing is muted or auto-cleared any more. Keep root_to_symptoms_/muted_faults_ keyed by FaultId so a clear still hits exact records, but match symptoms across owners by default. If owner scoping is wanted, make it an opt-in rule option (e.g. same_source: true).

// Check if we have a pending cluster for this rule AND this owner. One rule forms
// one cluster per owner, so a burst on owner A never counts owner B's faults
// towards min_count and never mutes them as non-representative members.
const auto pending_key = std::make_pair(rule.id, id.owner);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue for auto-cluster. With the owner in the key, a storm like MOTOR_COMM_FL from the motor node, LIDAR_TIMEOUT from the lidar and CAMERA_TIMEOUT from the camera becomes three one-member pending clusters and never reaches min_count. Key by rule.id only and store FaultId members instead of bare codes. ClusterInfo then needs the member owners too, otherwise representative_code cannot be resolved to a record (empty source_id is ambiguous).

if (fault) {
publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CLEARED, *fault, auto_cleared_codes);
audit_transition(kTransitionCleared, *fault, "clear_service", get_wall_clock_time().nanoseconds());
audit_transition(kTransitionCleared, *fault, target->owner, get_wall_clock_time().nanoseconds());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the owner as source on every row, the audit log can no longer tell an operator clear from the correlation cascade clear (L1060) or the startup HEALED->CLEARED reclassify (L281): all three are cleared with the same source. Same for timer vs report confirmation (L460 vs L902). Keep the owner, but record the mechanism (clear_service, correlation_auto_clear, startup_reclassify, auto_confirm_timer, auto_heal) in its own AuditEvent field.

row.last_occurred_ns = select.column_int64(4);
row.occurrence_count = select.column_int64(5);
row.status = select.column_text(6);
row.owner = legacy_first_source(select.column_text(7));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Legacy reporting_sources was written from a std::set, so the first entry is the alphabetically smallest source, not the one driving the status. The other sources' PASSED now hit a missing record and are ignored, so a migrated CONFIRMED fault can only heal if that one source reports again. At minimum log the dropped sources per fault so an operator knows to clear it by hand.

// has exactly ONE owner. The uncorrelated form this replaces picked whichever row
// SQLite reached first, which on a code with two owners handed one owner's
// snapshots and recordings to the other.
const std::string sql = std::string("UPDATE ") + table + " SET owner = (SELECT f.owner FROM faults f WHERE " +

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Child rows left with an empty owner are re-checked on every open, so once a new source becomes the only owner of that code, the next restart hands it old orphaned snapshots, freeze frames and rosbag links of an unrelated earlier fault. Run this backfill only inside the migration and mark leftovers with kLegacyOwner instead of ''.

}
}
auto it = last_capture_times_.find(fault_code);
auto it = last_capture_times_.find(id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cooldown is now per record, but SnapshotCapture resolves topics by code. N sources confirming one code enqueue N captures of the same topics, and with drop_oldest that evicts pending captures of other faults. Capture once per code within the cooldown and fan the rows out to each owner, or add a per-code cooldown next to the per-record one.

// several rows pointing at the same bag, and pushing the code once per row would
// repeat it. First-seen order is kept, so the codes still read in the order the
// recording collected them.
const auto attach_code = [&attached_codes](const std::string & code) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deduping to codes drops the owners, and that is why #700 has to call ListRosbags once per candidate owner to authorize a download. Add string[] owners next to fault_codes in GetRosbag.srv (one entry per row) and fill it here.

// thing being fixed.
SqliteStatement newest(db_, "SELECT MAX(capture_id) FROM snapshots WHERE fault_code = ?");
newest.bind_text(1, fault_code);
SqliteStatement newest(db_, "SELECT MAX(capture_id) FROM snapshots WHERE fault_code = ? AND owner = ?");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trim loop (MAX, COUNT, DELETE) now filters on (fault_code, owner), but the snapshots indexes are still (fault_code) and (fault_code, topic). Add an index on (fault_code, owner, capture_id), like you did for rosbag_files.

"DELETE FROM near_misses WHERE fault_code = ?1 AND id NOT IN "
"(SELECT id FROM near_misses WHERE fault_code = ?1 "
"ORDER BY id DESC LIMIT ?2)");
"DELETE FROM near_misses WHERE fault_code = ?1 AND source_id = ?2 AND id NOT IN "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This trim runs on every qualifying FAILED and now filters on source_id, but idx_near_misses_fault_code is still (fault_code, id), so it walks every owner's rows of the code. Widen it to (fault_code, source_id, id). get_near_misses (L1793) benefits too.

A fault record is identified by (fault_code, owner), where the owner is the
source_id the report arrived with. Status, debounce, severity, occurrence
count, freeze frames, snapshots, rosbag links, near misses and audit rows are
per record, and clears, heals and every per-record service call act on one
record. ClearFault, GetFault, GetSnapshots and GetRosbag gain a trailing
source_id; an empty one is refused as ambiguous when several records share
the code. MutedFaultInfo gains source_id. The SQLite store migrates on open.
The PostgreSQL backend identifies a record by (fault_code, owner) like the
SQLite one. owner is added to faults, freeze_frames, snapshots and
rosbag_files, the primary keys on the fault code become unique indexes that
include it, and every read, write, trim and delete is scoped to one record.
get_faults_by_code and the FaultId-keyed reclassify and time-based
confirmation come with it.

Tables written by the earlier layout are migrated in place on the first
connection, in one transaction: the owner is the first entry of a row's
reporting_sources (legacy when none is readable), pending rows lose their
confirmation time, and evidence rows are assigned only where their code has
exactly one owner. A stored CRITICAL severity confirms the record and a
reactivation clears its confirmation time, as in the SQLite backend.
… record

A fault code now has one record shared by every source that reports it,
as before the record identity change, unless its entry in
fault_thresholds.yaml sets `per_source: true`. That key is a strict YAML
boolean: any other value rejects the file like every other malformed value.

The shared record is owned by the reserved id `*`, which ReportFault
refuses as a source_id. One function, FaultManagerCore::record_id_for,
decides the identity of a report. The storage backends receive the decided
FaultId and take the reporting source separately, for reporting_sources and
the near-miss entries. A single-record request naming a source_id reaches
the record it owns, otherwise the record that source reports into; an empty
one keeps the unscoped resolution.

The shared record has one debounce counter again, so the warning for a code
debounced under two policies comes back for shared codes. A per-source
record never warns.

Stored records keep the identity they were written with when a code is
switched between modes. At startup the node logs one warning per code that
holds active records of the other identity.

A store written when every code was one record opens with each row as the
shared record of its code, keeping its reporting_sources. Its snapshots,
freeze frame, recording links and near misses attach to it, the near
misses of every source included. near_misses gains an owner column and is
read and trimmed per record. The owner guessed from the first source, and
the code that existed only for that guess, are gone. The PostgreSQL
migration does the same.

Entity-default snapshot and rosbag topic resolution reads the record's
reporting sources again, so a shared record captures the topics of every
node that reported it.
A shared record (default code) keeps the original rule: the severity of the
report decides critical_immediate_confirm. A per-source record decides on its
stored severity. The severity-mismatch warning is logged only for per-source
records.
Correlation rules match by fault code whoever reports it. A root cause of
one node mutes, and with auto_clear_with_root auto-clears, the matching
symptom records of any other node inside the window. An auto-cluster rule
forms one pending cluster that collects the matching records of every node.

The relations hold records: muted records, the root-to-symptom map, cluster
members and the cluster representative are FaultIds, so a clear acts on the
exact records the root cause muted and muting hides a record, not a code.

ClusterInfo gains fault_owners, one entry per fault_codes entry in the same
order, and representative_owner, so a consumer can address each member with
the single-record services.
The source of an audit row is again the report's source for a row a report
drove and the mechanism for an automatic one: auto_confirm_timer, auto_heal,
clear_service and startup_reclassify. A symptom cleared by the correlation
cascade is written as correlation_auto_clear, so it is told apart from an
operator clear.

AuditEvent gains a trailing owner: the owner of a per-source record, empty
for a shared record. canonicalize hashes it only when it is not empty, so
every row of a shared record and every existing chain hash as before. An
audit log created without the column gets it on open.
Per-source records of one code that confirm inside the recapture cooldown
share one capture of the code's topics. The first worker to start samples;
every record that joined while it ran, and every record that confirms after
it finished and before the cooldown ends, gets the same rows and the same
freeze frame under its own owner. The queue holds no job for a joining
record unless a recording still needs one, so a storm of sources does not
fill the capture queue with identical snapshot work.

A record that confirms again after it was acknowledged gets a capture of its
own. A shared record, a disabled cooldown, and a code whose topics come from
the reporting node (entity default) are captured per record as before.
GetRosbag gains a trailing owners array, parallel to fault_codes, with one
entry per fault record the recording is attached to: the reporting source
for a per-source record, `*` for a shared one. fault_codes is now one entry
per record too, so several per-source records of one code list the code once
per owner and the two arrays stay parallel. A shared code is still one entry.
The per-record snapshot trims and reads filter on (fault_code, owner), so
snapshots get an index on (fault_code, owner, capture_id) on SQLite and
PostgreSQL. The near-miss index on (fault_code, owner, id) was already per
record. An existing store gets both on open.
Opening a migrated SQLite or PostgreSQL store with an earlier release is
unsupported: the earlier release's startup cleanup deletes rosbag_files rows
that two owners of one code share. A copy taken before the upgrade is the only
way back; there is no guard in the code.
Every ClearFault the OPC UA plugin sends carries the entity id the fault is
reported under as source_id: the device alarm clear, the event alarm clear,
the link-state clear and the SOVD scoped clear. The comms-lost probe names it
too. For a code kept per source, a clear without it is refused as ambiguous as
soon as a second source reported the code, and the plugin never reads the
answer, so the fault stayed standing.
A code has at most one queued job for its group: a record joining a queued
capture queues nothing, so a storm of sources no longer fills the capture
queue and evicts other faults' captures. When that one job is evicted or
refused, the group's capture is lost for its members and the group goes, so
a later confirmation starts a new one.

Only finished groups age out with the cooldown. A queued or running group
lives until its capture completes, so a capture longer than the cooldown
still writes every record that joined it, and a group whose capture fails is
dropped.

A record that confirms again while the capture of its previous occurrence
still runs leaves that group and is captured alone, after the new
confirmation, instead of being given the old sample.
The plugin remembers the entity id under which it reported the standing
comms-lost fault, and the probe and the clear name that id until the clear
is sent; after it the current component id is used again. A per-source
record raised under the placeholder derived from the endpoint is therefore
cleared after the component is renamed, instead of being left standing.
Shared codes are unaffected.
The cascade clears the records the root cause has muted since its last
report, because a new report of the root cause starts that list again.
Fields added inside the elements of the ListFaults arrays change the wire
layout, so a client built against earlier messages misreads a response that
carries a muted or clustered entry without any error. The compatibility notes
say so for every service and topic of the fault manager.
The tests that origin/main had for a record several sources report into
are restored: UpdateExistingFaultPublishesUpdatedEvent and
GetFaultReturnsExtendedDataRecords report the second time from another
source, and the in-memory cases for confirmation, debounce depth, healing
and recurrence use the shared owner. The confirmed-latch tests use the
shared owner too: with a per-source owner the final report took the
stored-severity rule and never reached the latch.

A launch test on the in-memory and the SQLite store checks that two sources
reporting a default code give one confirmed event and one updated event
that names both sources with the occurrence count still 1.
…process

- An unscoped GetSnapshots or GetRosbag on a per-source code with two owners
  is refused as ambiguous and returns no evidence.
- A recapture cooldown of 0 turns snapshot grouping off: two owners confirmed
  back to back are two captures.
- With max_per_fault 1 and snapshots retained on clear, an owner that confirms
  twice keeps only its newest capture, both topics of it, and another owner's
  capture is intact.
- A cleared cluster member leaves the cluster, the representative moves off
  it, and the member re-reported later is in the cluster exactly once.
- A thresholds file with a non-bool per_source is rejected as a whole, so the
  code stays one shared record listing both sources.
…after a restart

After a restart the plugin remembers no reporting id, so the comms-lost
probe under the current component id found nothing for a per-source code and
the record raised under the stand-in derived from the endpoint stayed
standing. When the probe under the remembered or current id finds no record,
the plugin now probes under each other id this process reports under, and
clears the record it finds under the id that found it. A shared code is
found by the first probe, as before.
Two sources report one per-source code. The alarm bit of the bridge
falls, and the test requires the bridge's record to clear while the
other source's record stays confirmed. Without the source in the
ClearFault request the call is refused as ambiguous and the test fails.
…ascade flag says

Clearing a root cause with skip_correlation_auto_clear left its symptoms
muted and kept the root muting later reports, so the symptoms vanished
from the default listing under a root that no longer existed. The
correlation state of the root is now always released on clear; the flag
only decides whether the symptoms are cleared too.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the same defect existed there.
…muted

A report of a root cause that is still active reset its symptom list, so
its later clear cascaded only to the symptoms muted after the last
report. The report now renews the window only. A symptom is muted when it
is reported within window_ms of the latest report of the root cause, and
it stays muted until the root cause is cleared.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the same defect existed there.
…ted by the node

The test writes the old one-record-per-code layout with faults, a
snapshot, a freeze frame and a rosbag link, starts the real node on it,
and reads the records back through the services, before and after a
restart. A small helper executable runs the SQL because the test has no
PostgreSQL client. The test is built only with POSTGRES_SUPPORT and runs
in the CI postgres job.
… answer

The negative proof that an answer to a timed-out probe decides nothing
slept a fixed two seconds. It now waits until a further probe is parked,
which shows the decision ran again after the late answer arrived.
…overy

The first test raced the node's discovery of the published topics, and
compared capture times that two captures can share when the topics
publish slower than the captures run. A throw-away fault now waits until
a capture holds both topics, and the separation of two captures is
asserted by their capture ids alone.
A symptom that was cleared or ended stayed in the list of the root cause
that had muted it. When the root cause was reported again and the symptom
came back outside the window, the root cause's clear cascaded to that
later occurrence, which it never muted, and the list also grew for every
owner of a per-source code. A record that ends now leaves the list of
every root cause.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the same defect existed there.
Only the record a ClearFault named released its relations. A symptom
cleared by that cascade, and a root cause that reached HEALED, kept the
symptoms they had muted hidden under a root that no longer existed. A
cascade-cleared record and a healed record now end their relations as a
direct clear does: their symptoms are unmuted and keep their status.
Only a ClearFault without the skip flag cascades, one level.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the same defect existed there.
…ames none

The set of ids a bridge reports under always held the NodeMap default id,
although only a node map that names no component_id reports under it. A
bridge with an explicit component_id therefore treated a record held under
the default id as its own: it probed that id after its own probe missed,
and cleared a per-source PLC_COMMS_LOST record that belongs to a default-id
bridge. For a shared code the same set let it clear a default-id bridge's
record, a defect that exists on origin/main as well. A decision that finds
no record of the bridge now says so in the log.

This changes the behaviour of default (shared) codes against origin/main
on purpose.
…t knows

The test recorded topics "all", so the bag stayed empty only while the
whole ROS domain was silent. Another process on the domain filled the
bag with its traffic and the empty-bag assertion failed. The test now
records a topic nobody publishes on, with every assertion unchanged.
…presentative

The cluster only logged the members it meant to hide and never wrote them
into the muted set, so the default listing showed all of them. Members
other than the representative are now muted while the cluster is active,
on activation and on arrival. The set follows the representative: when it
changes, the new one is shown and the old one is muted, and when the shown
member is cleared the next representative is shown. A cluster ends when
its last member is cleared. Muting is display only: GetFault and the
storage see a muted member as they see an unmuted one, and a member that
stops being muted keeps its status.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the listing now matches what the documentation promised.
The ids a bridge reports under included the stand-ins derived from its
endpoint for every bridge. A bridge with a node map never reports under a
stand-in, and the derivation uses the host only, so two bridges on one
host collided: the bridge with a node map probed the stand-in of a
config-less bridge and cleared its PLC_COMMS_LOST record. The stand-ins
are now part of the set only when no node map is configured. A bridge
reconfigured from config-less to a node map no longer clears the record
it raised under its old stand-in.

This changes the behaviour of shared codes against origin/main on
purpose: main has the same loop.
A record had one muted entry, and ending a root cause erased the entry of
every symptom it listed, although another root cause that lists the same
symptom under a muting rule, or an active show_as_single cluster, still
muted it. The entry is now set from the relations that are still live
whenever one relation ends: a direct clear with or without the skip flag,
a cascade and a heal. A root of a rule with mute_symptoms false no longer
erases a mute it never wrote. A record that no relation mutes is
unmuted and keeps its status.

This changes the behaviour of default (shared) codes against origin/main
on purpose: the same defect existed there.
…where stays out

A publisher on the probe topic that never publishes and a publisher on an
unrelated topic now exist before the capture. The test waits until the
capture has subscribed to the probe topic, requires that it has not
subscribed to the unrelated one, and publishes on the unrelated topic
through the post-roll, so the empty bag is evidence of the topic
selection and not of a quiet domain.
…inds

The tutorial said the cascade clears exactly the records the root cause
muted. That holds for a rule with mute_symptoms true. A rule with
mute_symptoms false lists symptoms it does not mute, and its cascade
clears the listed records. The text now says what holds for both, and
that a record stays muted while any root cause or active cluster still
mutes it.
Reconciling the muted entry of a record looped over every hierarchical
rule that matched both codes, so a later rule that the root is not bound
to could mute a symptom whose real relation does not mute. A root is
bound to the first rule its code matches, and only that rule decides
whether it mutes. The delay of a symptom behind a root is now kept per
relation, recorded when the record last attached to that root and erased
with the relation, so a mute that moves to another root carries that
root's delay.
@bburda
bburda force-pushed the feat/fault-identity-owner-key branch from 234acc5 to 7db52f6 Compare November 13, 2026 15:25
@bburda bburda changed the title fault_manager: key every fault record by fault code and owner fault_manager: optional per-source fault records, chosen per fault code Nov 13, 2026
Subscription.get_publisher_count does not exist in the Humble rclpy.
Node.count_publishers does on every distro.
The program overrode the capture hooks and called the storage and the
core with a bare fault code. It now passes the record id to the hooks and
the storage, and the source id to the core's clear. Every check it makes
is kept.
…re reporting

The grouping and queue tests reported a fault at once after creating their
publisher. A fault manager that had not discovered the topic yet skipped
the snapshot and the first test failed. A snapshot pattern for probe codes
on the published topic lets setUpClass report fresh probe codes until one
lists a snapshot, so the tests start after discovery.
…c was not discovered

The silent topic is created before the probed one, so the fault manager
has processed it once it has discovered the probed topic. After shutdown
the test requires that the fault manager never logged the silent topic as
not found, because every capture that skipped it at once would leave
nothing busy and the tests would pass without exercising the queue.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants