Repository navigation
Conversation
| for (const auto & prc : pending_root_causes_) { | ||
| // A root cause only explains its OWN reporter's faults. Without this the first | ||
| // owner to report a root-cause code would mute every other owner's symptom. | ||
| if (prc.fault_id.owner != id.owner) { |
There was a problem hiding this comment.
This owner check (and the (rule_id, owner) cluster key at line 437) turns every hierarchical and cluster relation into a same-owner relation, which #698 does not ask for and which silently retires the documented cross-node behaviour: the tutorial's E-stop cascade and sensor storm span nodes, docs/tutorials/fault-correlation.rst still explains skip_correlation_auto_clear as the guard against cascading into "apps in other entities", and the fault_manager integration tests only pass because the symptom source was rewritten from /motor_node to /estop_node (test_integration.test.py:814-816) and the storm from /node{i + 1} to /node1. Keep matching by code across owners and make the relation per record (mute and cascade each symptom FaultId on its own, which is what "muting hides a record rather than a code" means), or if same-owner correlation is the intended design, call it out as a breaking change in the PR and update the tutorial.
| The rebuild is one way. A fault manager from an earlier release opened on a migrated store aborts | ||
| on the first report of a fault code the store does not hold yet | ||
| (`NOT NULL constraint failed: faults.owner`), and on a fault code two sources share it writes one | ||
| source's data into the other source's row. If a rollback may be needed, keep a copy of `faults.db` |
There was a problem hiding this comment.
"Aborts on the first report of a fault code the store does not hold yet" understates the downgrade: main's initialize_schema runs DELETE FROM rosbag_files WHERE id NOT IN (SELECT MAX(id) FROM rosbag_files GROUP BY fault_code, file_path) on open, which silently drops one owner's link row for every bag two owners of a code share (the burst case finalize_post_fault_recording now produces), before any report and before the NOT NULL abort, so an earlier release opened on a migrated store loses rows quietly rather than failing loudly. Either guard the old code path (a faults view over a renamed table makes main's startup UPDATE faults SET last_occurred_ns ... throw before it writes anything) or state here that opening a migrated file with an earlier release deletes rows and is unsupported, with the backup as the only way back.
| # Reporting source that owns the fault record (the source_id used in ReportFault). | ||
| # Together with fault_code it identifies one record. Empty: the call applies only | ||
| # when exactly one record carries fault_code, otherwise it fails as ambiguous. | ||
| string source_id |
There was a problem hiding this comment.
The in-tree OPC UA plugin still calls this service without the new field: OpcuaPlugin::send_clear_fault (src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp:1284, reached from on_alarm_change, on_event_alarm and the FaultProvider clear_fault) sends only fault_code, while its reports carry entity_id as source_id. PLC_COMMS_LOST is raised per component, so once a second emitter of that code has ever reported to the same fault_manager the plugin's clear is refused as ambiguous (cleared records count too) and, since the plugin never reads the response, the fault stays CONFIRMED with no error on the plugin side. Pass the entity id through as request->source_id at the three call sites.
| /// Pending cluster with steady_clock timestamp for window tracking | ||
| struct PendingCluster { | ||
| ClusterData data; | ||
| std::string owner; ///< Every member record of this cluster has this owner |
There was a problem hiding this comment.
Clusters are per owner now, but ClusterInfo.msg still carries bare fault_codes and no owner, so a consumer of ListFaults(include_clusters=true) can neither address the member records with the single-record services nor tell two clusters of one rule apart beyond the counter suffix in cluster_id. MutedFaultInfo got source_id for exactly this reason - give ClusterInfo the same field, filled from PendingCluster::owner.
| auto fault = storage_->get_fault(reclassified_id); | ||
| if (fault) { | ||
| audit_transition(kTransitionCleared, *fault, "startup_reclassify", reclassified_at_ns); | ||
| audit_transition(kTransitionCleared, *fault, reclassified_id.owner, reclassified_at_ns); |
There was a problem hiding this comment.
Writing the owner into source for the automatic transitions (here, the timer at line 460, the heal at 905, the cascade at 1060) drops information the chain used to carry: main wrote startup_reclassify, auto_confirm_timer, auto_heal and clear_service, and now a cleared row from an operator's ClearFault, a startup reclassification and a correlation cascade are byte-identical, as are confirmed by report and by timer, which the README's new claim that "the transition column already says what moved the row" (README.md:253) does not cover. Keep the driver in the row alongside the owner, for example a new field folded into canonicalize only when non-empty so existing chains still verify, and correct the README sentence.
| for (const auto & prc : pending_root_causes_) { | ||
| // A root cause only explains its OWN reporter's faults. Without this the first | ||
| // owner to report a root-cause code would mute every other owner's symptom. | ||
| if (prc.fault_id.owner != id.owner) { |
There was a problem hiding this comment.
This drops cross-node correlation, which is the main use case. In the README estop_cascade example ESTOP_001 comes from the safety node and MOTOR_COMM_* from the motor controllers, so with this check nothing is muted or auto-cleared any more. Keep root_to_symptoms_/muted_faults_ keyed by FaultId so a clear still hits exact records, but match symptoms across owners by default. If owner scoping is wanted, make it an opt-in rule option (e.g. same_source: true).
| // Check if we have a pending cluster for this rule AND this owner. One rule forms | ||
| // one cluster per owner, so a burst on owner A never counts owner B's faults | ||
| // towards min_count and never mutes them as non-representative members. | ||
| const auto pending_key = std::make_pair(rule.id, id.owner); |
There was a problem hiding this comment.
Same issue for auto-cluster. With the owner in the key, a storm like MOTOR_COMM_FL from the motor node, LIDAR_TIMEOUT from the lidar and CAMERA_TIMEOUT from the camera becomes three one-member pending clusters and never reaches min_count. Key by rule.id only and store FaultId members instead of bare codes. ClusterInfo then needs the member owners too, otherwise representative_code cannot be resolved to a record (empty source_id is ambiguous).
| if (fault) { | ||
| publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CLEARED, *fault, auto_cleared_codes); | ||
| audit_transition(kTransitionCleared, *fault, "clear_service", get_wall_clock_time().nanoseconds()); | ||
| audit_transition(kTransitionCleared, *fault, target->owner, get_wall_clock_time().nanoseconds()); |
There was a problem hiding this comment.
With the owner as source on every row, the audit log can no longer tell an operator clear from the correlation cascade clear (L1060) or the startup HEALED->CLEARED reclassify (L281): all three are cleared with the same source. Same for timer vs report confirmation (L460 vs L902). Keep the owner, but record the mechanism (clear_service, correlation_auto_clear, startup_reclassify, auto_confirm_timer, auto_heal) in its own AuditEvent field.
| row.last_occurred_ns = select.column_int64(4); | ||
| row.occurrence_count = select.column_int64(5); | ||
| row.status = select.column_text(6); | ||
| row.owner = legacy_first_source(select.column_text(7)); |
There was a problem hiding this comment.
Legacy reporting_sources was written from a std::set, so the first entry is the alphabetically smallest source, not the one driving the status. The other sources' PASSED now hit a missing record and are ignored, so a migrated CONFIRMED fault can only heal if that one source reports again. At minimum log the dropped sources per fault so an operator knows to clear it by hand.
| // has exactly ONE owner. The uncorrelated form this replaces picked whichever row | ||
| // SQLite reached first, which on a code with two owners handed one owner's | ||
| // snapshots and recordings to the other. | ||
| const std::string sql = std::string("UPDATE ") + table + " SET owner = (SELECT f.owner FROM faults f WHERE " + |
There was a problem hiding this comment.
Child rows left with an empty owner are re-checked on every open, so once a new source becomes the only owner of that code, the next restart hands it old orphaned snapshots, freeze frames and rosbag links of an unrelated earlier fault. Run this backfill only inside the migration and mark leftovers with kLegacyOwner instead of ''.
| } | ||
| } | ||
| auto it = last_capture_times_.find(fault_code); | ||
| auto it = last_capture_times_.find(id); |
There was a problem hiding this comment.
The cooldown is now per record, but SnapshotCapture resolves topics by code. N sources confirming one code enqueue N captures of the same topics, and with drop_oldest that evicts pending captures of other faults. Capture once per code within the cooldown and fan the rows out to each owner, or add a per-code cooldown next to the per-record one.
| // several rows pointing at the same bag, and pushing the code once per row would | ||
| // repeat it. First-seen order is kept, so the codes still read in the order the | ||
| // recording collected them. | ||
| const auto attach_code = [&attached_codes](const std::string & code) { |
There was a problem hiding this comment.
Deduping to codes drops the owners, and that is why #700 has to call ListRosbags once per candidate owner to authorize a download. Add string[] owners next to fault_codes in GetRosbag.srv (one entry per row) and fill it here.
| // thing being fixed. | ||
| SqliteStatement newest(db_, "SELECT MAX(capture_id) FROM snapshots WHERE fault_code = ?"); | ||
| newest.bind_text(1, fault_code); | ||
| SqliteStatement newest(db_, "SELECT MAX(capture_id) FROM snapshots WHERE fault_code = ? AND owner = ?"); |
There was a problem hiding this comment.
The trim loop (MAX, COUNT, DELETE) now filters on (fault_code, owner), but the snapshots indexes are still (fault_code) and (fault_code, topic). Add an index on (fault_code, owner, capture_id), like you did for rosbag_files.
| "DELETE FROM near_misses WHERE fault_code = ?1 AND id NOT IN " | ||
| "(SELECT id FROM near_misses WHERE fault_code = ?1 " | ||
| "ORDER BY id DESC LIMIT ?2)"); | ||
| "DELETE FROM near_misses WHERE fault_code = ?1 AND source_id = ?2 AND id NOT IN " |
There was a problem hiding this comment.
This trim runs on every qualifying FAILED and now filters on source_id, but idx_near_misses_fault_code is still (fault_code, id), so it walks every owner's rows of the code. Widen it to (fault_code, source_id, id). get_near_misses (L1793) benefits too.
A fault record is identified by (fault_code, owner), where the owner is the source_id the report arrived with. Status, debounce, severity, occurrence count, freeze frames, snapshots, rosbag links, near misses and audit rows are per record, and clears, heals and every per-record service call act on one record. ClearFault, GetFault, GetSnapshots and GetRosbag gain a trailing source_id; an empty one is refused as ambiguous when several records share the code. MutedFaultInfo gains source_id. The SQLite store migrates on open.
The PostgreSQL backend identifies a record by (fault_code, owner) like the SQLite one. owner is added to faults, freeze_frames, snapshots and rosbag_files, the primary keys on the fault code become unique indexes that include it, and every read, write, trim and delete is scoped to one record. get_faults_by_code and the FaultId-keyed reclassify and time-based confirmation come with it. Tables written by the earlier layout are migrated in place on the first connection, in one transaction: the owner is the first entry of a row's reporting_sources (legacy when none is readable), pending rows lose their confirmation time, and evidence rows are assigned only where their code has exactly one owner. A stored CRITICAL severity confirms the record and a reactivation clears its confirmation time, as in the SQLite backend.
… record A fault code now has one record shared by every source that reports it, as before the record identity change, unless its entry in fault_thresholds.yaml sets `per_source: true`. That key is a strict YAML boolean: any other value rejects the file like every other malformed value. The shared record is owned by the reserved id `*`, which ReportFault refuses as a source_id. One function, FaultManagerCore::record_id_for, decides the identity of a report. The storage backends receive the decided FaultId and take the reporting source separately, for reporting_sources and the near-miss entries. A single-record request naming a source_id reaches the record it owns, otherwise the record that source reports into; an empty one keeps the unscoped resolution. The shared record has one debounce counter again, so the warning for a code debounced under two policies comes back for shared codes. A per-source record never warns. Stored records keep the identity they were written with when a code is switched between modes. At startup the node logs one warning per code that holds active records of the other identity. A store written when every code was one record opens with each row as the shared record of its code, keeping its reporting_sources. Its snapshots, freeze frame, recording links and near misses attach to it, the near misses of every source included. near_misses gains an owner column and is read and trimmed per record. The owner guessed from the first source, and the code that existed only for that guess, are gone. The PostgreSQL migration does the same. Entity-default snapshot and rosbag topic resolution reads the record's reporting sources again, so a shared record captures the topics of every node that reported it.
A shared record (default code) keeps the original rule: the severity of the report decides critical_immediate_confirm. A per-source record decides on its stored severity. The severity-mismatch warning is logged only for per-source records.
Correlation rules match by fault code whoever reports it. A root cause of one node mutes, and with auto_clear_with_root auto-clears, the matching symptom records of any other node inside the window. An auto-cluster rule forms one pending cluster that collects the matching records of every node. The relations hold records: muted records, the root-to-symptom map, cluster members and the cluster representative are FaultIds, so a clear acts on the exact records the root cause muted and muting hides a record, not a code. ClusterInfo gains fault_owners, one entry per fault_codes entry in the same order, and representative_owner, so a consumer can address each member with the single-record services.
The source of an audit row is again the report's source for a row a report drove and the mechanism for an automatic one: auto_confirm_timer, auto_heal, clear_service and startup_reclassify. A symptom cleared by the correlation cascade is written as correlation_auto_clear, so it is told apart from an operator clear. AuditEvent gains a trailing owner: the owner of a per-source record, empty for a shared record. canonicalize hashes it only when it is not empty, so every row of a shared record and every existing chain hash as before. An audit log created without the column gets it on open.
Per-source records of one code that confirm inside the recapture cooldown share one capture of the code's topics. The first worker to start samples; every record that joined while it ran, and every record that confirms after it finished and before the cooldown ends, gets the same rows and the same freeze frame under its own owner. The queue holds no job for a joining record unless a recording still needs one, so a storm of sources does not fill the capture queue with identical snapshot work. A record that confirms again after it was acknowledged gets a capture of its own. A shared record, a disabled cooldown, and a code whose topics come from the reporting node (entity default) are captured per record as before.
GetRosbag gains a trailing owners array, parallel to fault_codes, with one entry per fault record the recording is attached to: the reporting source for a per-source record, `*` for a shared one. fault_codes is now one entry per record too, so several per-source records of one code list the code once per owner and the two arrays stay parallel. A shared code is still one entry.
The per-record snapshot trims and reads filter on (fault_code, owner), so snapshots get an index on (fault_code, owner, capture_id) on SQLite and PostgreSQL. The near-miss index on (fault_code, owner, id) was already per record. An existing store gets both on open.
Opening a migrated SQLite or PostgreSQL store with an earlier release is unsupported: the earlier release's startup cleanup deletes rosbag_files rows that two owners of one code share. A copy taken before the upgrade is the only way back; there is no guard in the code.
Every ClearFault the OPC UA plugin sends carries the entity id the fault is reported under as source_id: the device alarm clear, the event alarm clear, the link-state clear and the SOVD scoped clear. The comms-lost probe names it too. For a code kept per source, a clear without it is refused as ambiguous as soon as a second source reported the code, and the plugin never reads the answer, so the fault stayed standing.
A code has at most one queued job for its group: a record joining a queued capture queues nothing, so a storm of sources no longer fills the capture queue and evicts other faults' captures. When that one job is evicted or refused, the group's capture is lost for its members and the group goes, so a later confirmation starts a new one. Only finished groups age out with the cooldown. A queued or running group lives until its capture completes, so a capture longer than the cooldown still writes every record that joined it, and a group whose capture fails is dropped. A record that confirms again while the capture of its previous occurrence still runs leaves that group and is captured alone, after the new confirmation, instead of being given the old sample.
The plugin remembers the entity id under which it reported the standing comms-lost fault, and the probe and the clear name that id until the clear is sent; after it the current component id is used again. A per-source record raised under the placeholder derived from the endpoint is therefore cleared after the component is renamed, instead of being left standing. Shared codes are unaffected.
The cascade clears the records the root cause has muted since its last report, because a new report of the root cause starts that list again.
Fields added inside the elements of the ListFaults arrays change the wire layout, so a client built against earlier messages misreads a response that carries a muted or clustered entry without any error. The compatibility notes say so for every service and topic of the fault manager.
The tests that origin/main had for a record several sources report into are restored: UpdateExistingFaultPublishesUpdatedEvent and GetFaultReturnsExtendedDataRecords report the second time from another source, and the in-memory cases for confirmation, debounce depth, healing and recurrence use the shared owner. The confirmed-latch tests use the shared owner too: with a per-source owner the final report took the stored-severity rule and never reached the latch. A launch test on the in-memory and the SQLite store checks that two sources reporting a default code give one confirmed event and one updated event that names both sources with the occurrence count still 1.
…process - An unscoped GetSnapshots or GetRosbag on a per-source code with two owners is refused as ambiguous and returns no evidence. - A recapture cooldown of 0 turns snapshot grouping off: two owners confirmed back to back are two captures. - With max_per_fault 1 and snapshots retained on clear, an owner that confirms twice keeps only its newest capture, both topics of it, and another owner's capture is intact. - A cleared cluster member leaves the cluster, the representative moves off it, and the member re-reported later is in the cluster exactly once. - A thresholds file with a non-bool per_source is rejected as a whole, so the code stays one shared record listing both sources.
…after a restart After a restart the plugin remembers no reporting id, so the comms-lost probe under the current component id found nothing for a per-source code and the record raised under the stand-in derived from the endpoint stayed standing. When the probe under the remembered or current id finds no record, the plugin now probes under each other id this process reports under, and clears the record it finds under the id that found it. A shared code is found by the first probe, as before.
Two sources report one per-source code. The alarm bit of the bridge falls, and the test requires the bridge's record to clear while the other source's record stays confirmed. Without the source in the ClearFault request the call is refused as ambiguous and the test fails.
…ascade flag says Clearing a root cause with skip_correlation_auto_clear left its symptoms muted and kept the root muting later reports, so the symptoms vanished from the default listing under a root that no longer existed. The correlation state of the root is now always released on clear; the flag only decides whether the symptoms are cleared too. This changes the behaviour of default (shared) codes against origin/main on purpose: the same defect existed there.
…muted A report of a root cause that is still active reset its symptom list, so its later clear cascaded only to the symptoms muted after the last report. The report now renews the window only. A symptom is muted when it is reported within window_ms of the latest report of the root cause, and it stays muted until the root cause is cleared. This changes the behaviour of default (shared) codes against origin/main on purpose: the same defect existed there.
…ted by the node The test writes the old one-record-per-code layout with faults, a snapshot, a freeze frame and a rosbag link, starts the real node on it, and reads the records back through the services, before and after a restart. A small helper executable runs the SQL because the test has no PostgreSQL client. The test is built only with POSTGRES_SUPPORT and runs in the CI postgres job.
… answer The negative proof that an answer to a timed-out probe decides nothing slept a fixed two seconds. It now waits until a further probe is parked, which shows the decision ran again after the late answer arrived.
…overy The first test raced the node's discovery of the published topics, and compared capture times that two captures can share when the topics publish slower than the captures run. A throw-away fault now waits until a capture holds both topics, and the separation of two captures is asserted by their capture ids alone.
A symptom that was cleared or ended stayed in the list of the root cause that had muted it. When the root cause was reported again and the symptom came back outside the window, the root cause's clear cascaded to that later occurrence, which it never muted, and the list also grew for every owner of a per-source code. A record that ends now leaves the list of every root cause. This changes the behaviour of default (shared) codes against origin/main on purpose: the same defect existed there.
Only the record a ClearFault named released its relations. A symptom cleared by that cascade, and a root cause that reached HEALED, kept the symptoms they had muted hidden under a root that no longer existed. A cascade-cleared record and a healed record now end their relations as a direct clear does: their symptoms are unmuted and keep their status. Only a ClearFault without the skip flag cascades, one level. This changes the behaviour of default (shared) codes against origin/main on purpose: the same defect existed there.
…ames none The set of ids a bridge reports under always held the NodeMap default id, although only a node map that names no component_id reports under it. A bridge with an explicit component_id therefore treated a record held under the default id as its own: it probed that id after its own probe missed, and cleared a per-source PLC_COMMS_LOST record that belongs to a default-id bridge. For a shared code the same set let it clear a default-id bridge's record, a defect that exists on origin/main as well. A decision that finds no record of the bridge now says so in the log. This changes the behaviour of default (shared) codes against origin/main on purpose.
…t knows The test recorded topics "all", so the bag stayed empty only while the whole ROS domain was silent. Another process on the domain filled the bag with its traffic and the empty-bag assertion failed. The test now records a topic nobody publishes on, with every assertion unchanged.
…presentative The cluster only logged the members it meant to hide and never wrote them into the muted set, so the default listing showed all of them. Members other than the representative are now muted while the cluster is active, on activation and on arrival. The set follows the representative: when it changes, the new one is shown and the old one is muted, and when the shown member is cleared the next representative is shown. A cluster ends when its last member is cleared. Muting is display only: GetFault and the storage see a muted member as they see an unmuted one, and a member that stops being muted keeps its status. This changes the behaviour of default (shared) codes against origin/main on purpose: the listing now matches what the documentation promised.
The ids a bridge reports under included the stand-ins derived from its endpoint for every bridge. A bridge with a node map never reports under a stand-in, and the derivation uses the host only, so two bridges on one host collided: the bridge with a node map probed the stand-in of a config-less bridge and cleared its PLC_COMMS_LOST record. The stand-ins are now part of the set only when no node map is configured. A bridge reconfigured from config-less to a node map no longer clears the record it raised under its old stand-in. This changes the behaviour of shared codes against origin/main on purpose: main has the same loop.
A record had one muted entry, and ending a root cause erased the entry of every symptom it listed, although another root cause that lists the same symptom under a muting rule, or an active show_as_single cluster, still muted it. The entry is now set from the relations that are still live whenever one relation ends: a direct clear with or without the skip flag, a cascade and a heal. A root of a rule with mute_symptoms false no longer erases a mute it never wrote. A record that no relation mutes is unmuted and keeps its status. This changes the behaviour of default (shared) codes against origin/main on purpose: the same defect existed there.
…where stays out A publisher on the probe topic that never publishes and a publisher on an unrelated topic now exist before the capture. The test waits until the capture has subscribed to the probe topic, requires that it has not subscribed to the unrelated one, and publishes on the unrelated topic through the post-roll, so the empty bag is evidence of the topic selection and not of a quiet domain.
…inds The tutorial said the cascade clears exactly the records the root cause muted. That holds for a rule with mute_symptoms true. A rule with mute_symptoms false lists symptoms it does not mute, and its cascade clears the listed records. The text now says what holds for both, and that a record stays muted while any root cause or active cluster still mutes it.
Reconciling the muted entry of a record looped over every hierarchical rule that matched both codes, so a later rule that the root is not bound to could mute a symptom whose real relation does not mute. A root is bound to the first rule its code matches, and only that rule decides whether it mutes. The delay of a symptom behind a root is now kept per relation, recorded when the record last attached to that root and erased with the relation, so a mute that moves to another root carries that root's delay.
234acc5 to
7db52f6
Compare
Subscription.get_publisher_count does not exist in the Humble rclpy. Node.count_publishers does on every distro.
The program overrode the capture hooks and called the storage and the core with a bare fault code. It now passes the record id to the hooks and the storage, and the source id to the core's clear. Every check it makes is kept.
…re reporting The grouping and queue tests reported a fault at once after creating their publisher. A fault manager that had not discovered the topic yet skipped the snapshot and the first test failed. A snapshot pattern for probe codes on the published topic lets setUpClass report fresh probe codes until one lists a snapshot, so the tests start after discovery.
…c was not discovered The silent topic is created before the probed one, so the fault manager has processed it once it has discovered the probed topic. After shutdown the test requires that the fault manager never logged the silent topic as not found, because every capture that skipped it at once would leave nothing busy and the tests would pass without exercising the queue.
Pull Request
Summary
A fault code can now keep one record per reporting source. The choice is made per code:
reporting_sourceslistsevery source, and a clear or get without a source works.
per_source: trueon the code's entry infault_thresholds.yamlgives each source its own record, with its ownstatus, debounce, severity, occurrences, freeze frame, snapshots, rosbag links and near misses.
ClearFault,GetFault,GetSnapshotsandGetRosbagtake a trailingsource_id; an empty one is refused asambiguous:whenseveral records share the code. A code whose mode changes keeps its stored records and logs a warning at startup.
Also in this PR:
the symptoms it muted, also with
skip_correlation_auto_clear; a re-reported root keeps them;show_as_singlehidesthe members other than the current representative, as the tutorial describes. A record stays muted while another
root cause or an active cluster still mutes it. These fix behaviour that main has for every code.
the store before upgrading.
records under host-derived ids as its own, so two bridges on one host do not clear each other's records.
MutedFaultInfo,ClusterInfoandGetRosbaggain owner fields. Every client of the fault manager has to berebuilt against these messages.
Gateway support for per-source records is the next PR in this stack.
Issue
Type
Testing
src/ros2_medkit_fault_manager/test/integration/test_fault_identity.test.pyshows both modes over the services: adefault code reported by three sources is one record, a
per_sourcecode is three. PostgreSQL: build with-DPOSTGRES_SUPPORT=ONand setROS2_MEDKIT_TEST_PG_CONN(the CIpostgresjob does both).Checklist