Skip to content

gateway: address the fault record an entity owns - #700

Draft
bburda wants to merge 39 commits into
feat/fault-identity-owner-keyfrom
feat/fault-identity-gateway-owner
Draft

bburda wants to merge 39 commits into
feat/fault-identity-owner-keyfrom
feat/fault-identity-gateway-owner

Conversation

@bburda

@bburda bburda commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

With fault records keyed by (fault_code, owner) in the fault manager, the gateway still addressed faults by
code. This PR carries the owner through every per-record call and resolves each per-entity fault route to the
record the entity owns. It also closes the gaps that per-record faults exposed: recording downloads, plugin
clears, trigger reads and the event stream all act on one record, not on a code.

Stacked on #699, the fault manager half. Review that one first. After it merges this branch is rebased onto main.

Issue

Type

  • New feature or tests
  • Breaking change (per-entity routes answer 409 for an ambiguous code, plugin API version 8 rejects
    plugins built against version 7)

What changes

  • The transport and manager send source_id on every per-record call. The flat fault item carries
    source_id (the record owner), the detail carries x-medkit.owner and reporting_sources == [owner]. The
    list-level x-medkit.source_id keeps its existing meaning, the entity's namespace path.
  • A per-entity GET, DELETE or recording download addressed by code resolves in tiers: the records the entity's
    default fault list shows, then its muted records, then its cleared or healed ones. The first tier that holds
    the code decides. One record is acted on, several answer 409 with the vendor error code
    x-medkit-ambiguous-fault and the owners in parameters.owners, and the message points at the owning app's
    route. So a record a client read off the list is never ambiguous, and a muted or cleared record stays
    addressable. There is no global /faults/{code} route.
  • Per-entity and global DELETE-all clear each listed record with its owner and honour the entity lock on
    external bare ids. As before, a muted record is not cleared by DELETE-all, only by its per-code DELETE.
  • A plugin-served entity's clear carries the resolved owner to the plugin, and answers 503 when the fault store
    cannot be read instead of calling the plugin with nothing resolved.
  • Freeze frames are keyed per record and catch-up is deduped per record. The SSE stream coalesces updates per
    record, not per code, and hints an external component's bare id as entity_type: components.
  • A recording downloaded by its id is served to an entity only when one of the (fault code, owner) records it
    is attached to has its owner in the entity's fault scope, so two sources of one code never read each other's
    recordings. A compatibility URL carrying a bare code that addresses several records answers 409 like the
    fault routes. The download routes declare that 409, and its message names where a recording id is published.
  • The fault-trigger engine and trigger validation read an entity's values through its DataProvider first, so a
    rule on a provider-served entity fires. A provider's list envelope carries connected and timestamp, so the
    link-down guard applies there too. The trigger engine's own 409 states its rule.
  • Plugin API version 8: FaultProvider keeps clear_fault(entity_id, fault_code) and gains
    clear_fault_record(entity_id, fault_code, owner), whose default calls the two-argument clear. Plugin source
    written against version 7 compiles unchanged, and a pre-built version 7 binary is rejected by the loader.
    GatewayPlugin gains a protected log_sink() accessor.
  • OPC UA plugin: clears the record it raised, logs a refused clear from a reply callback that holds no pointer
    to the plugin, serves connected and timestamp in its data list, and answers its status route only for its
    own component (404 for any other component in the same gateway, as its data route already did).
  • OpenAPI, REST reference, locking and messages docs, tutorials and READMEs describe per-record faults.

Testing

  • The whole workspace the way CI runs it: 5003 unit and 1583 integration tests, no failure.
  • ASan with UBSan and TSan over the fault manager, gateway, OPC UA, fault reporter, diagnostic bridge and
    integration suites, no sanitizer report. No compiler warning beyond main's.
  • Lint, clang-tidy on the changed files, and a docs build with no new warning.
  • A scripted scenario run against main and against this branch (two owners of one code, a muted symptom,
    recordings on, one owner clearing): every difference in the API answers and the store is the intended one.
    Each app reaches its own record and recordings and gets 404 on the other's, the hosting component reaches
    both, and after one owner clears, the component's per-code routes act on the remaining record.
  • An upgraded store from main is served with every record.
  • Every new test was shown to fail on its own defect before the fix.

Checklist

  • Breaking changes are clearly described
  • Tests were added or updated
  • Docs were updated

Notes for the maintainer

  • When every record of a code in an entity's scope is cleared and there are several, the per-code route answers
    409 naming their owners. A client addresses one through its owner's app route.
  • The gateway README documents GET /api/v1/faults/{fault_code}/snapshots routes that no route registers
    (pre-existing).

A fault record is now the pair (fault_code, reporting source). The source
is the source_id a ReportFault call carried, and it owns the record.

ClearFault, GetFault, GetSnapshots and GetRosbag gain a trailing request
field source_id naming that owner. Empty means unscoped: the call applies
only when exactly one record carries the fault_code, and fails as
ambiguous when several do. On GetRosbag the field scopes the fault_code
lookup only, the recording_id path is unchanged.

Fault.msg and ReportFault.srv do not change shape. Their header comments,
FaultEvent.msg, ListFaultsForEntity.srv and the package README are
rewritten to describe the per-record model instead of aggregation by code.
A fault record is now identified by the pair (fault_code, owner), where the
owner is the source_id of the ReportFault call that created it. Two sources
reporting one code are two records: status, debounce counter, occurrence
count, timestamps, severity, freeze frame, snapshots, near misses, rosbag
links and capture cooldown are all per record, and a clear or a heal driven
by one owner never touches another owner's record.

Storage: FaultId addresses a record, FaultState carries its owner and emits
it as the single entry of reporting_sources, and the virtual API takes a
FaultId wherever it used to take a bare code. get_faults_by_code is new and
backs the unscoped service resolution. check_time_based_confirmation and
reclassify_healed_as_cleared return identities, because one code can hold a
moved record for one owner and an untouched one for another.

SQLite: faults gains owner with a composite unique index in place of the
single-column primary key, freeze_frames is rebuilt the same way, snapshots
and rosbag_files gain owner, and the rosbag unique index widens. A legacy
database is migrated by an explicit rebuild under one transaction, probing
each table on its own and backfilling the owner from the first entry of the
old reporting_sources array, which is the only per-source fact the old
schema recorded. Re-opening a migrated database changes nothing.

Node: ClearFault, GetFault, GetSnapshots and GetRosbag resolve their target
record from source_id, or unscoped when exactly one record carries the code,
failing with an ambiguous: message that lists the owners otherwise. Every
audit row now names the record's owner, so the clear_service, auto_heal,
auto_confirm_timer and startup_reclassify literals are gone. Capture,
snapshot writing and rosbag linking are per record, and correlation forms
every relation between records of one owner.

The unit tests that pinned one record per code are rewritten to the record
contract rather than deleted, and the new cases cover two owners of one
code, the unscoped resolution, the legacy-database migration and the burst
that links one recording to two records.
The integration cases that reported one fault_code from several sources
pinned the shared-record model: one row, one counter, one clear. They now
assert what the wire actually carries.

Two sources reporting one code are two records with their own severity,
description and occurrence count. Debounce depth is what one reporter has
said, so the cases that built a counter out of reports from several sources
now report from one. A PASSED is addressed to the record its own source
owns. A scoped clear takes one record and leaves the other CONFIRMED, an
unscoped clear of two is refused with an ambiguous message and changes
nothing, and a correlation cascade auto-clears only the symptom record of
the owner whose root cause was cleared.
Every sentence that said one fault_code is one fault, that the debounce
counter is shared by every source of a code, or that a fault tracks all its
reporting sources is rewritten to the record model: one record per
(fault_code, reporting source), each with its own state, each cleared on
its own.

Covers the fault manager README, its design page, the fault manager and
snapshot config comments, the message reference and the fault manager
configuration page, including how source_id scopes the single-record
services and what an unscoped call does when several records carry the code.
…r migration

Muting is a property of a record, not of a fault code. ListFaults built a set of
muted codes and erased every fault carrying one, so a second owner's un-muted
CONFIRMED record disappeared from the default view because the first owner's
identical code was muted under their root cause. The node now filters with the
engine's per-record is_muted, and muted_faults keeps its shape with one entry per
muted record.

GetRosbag reported fault_codes one entry per rosbag_files row, so a recording
shared by two owners of one code repeated that code. The codes are deduplicated
in first-seen order, which is what a caller authorizing a download needs.

The owner migration no longer refuses a database over the content of one row. It
ran json_extract over reporting_sources, and an earlier serializer escaped only
the quote, the backslash and \b \f \n \r \t, so a source_id carrying any other
control byte was written as text no JSON parser accepts. The rebuild then rolled
back and the node failed to start on every restart. The owner is now recovered in
C++ from the raw text, a row from which nothing can be read keeps an empty owner
and one warning, and new rows escape every byte below 0x20 as \u00XX so the
column is always valid JSON. The rebuild drops its scratch tables first, so
debris from an interrupted run or a manual repair cannot block every later open.

Evidence rows take the owner of their fault code only when that code has exactly
one owner. The uncorrelated subquery this replaces picked an arbitrary owner on a
shared code, which handed one owner another owner's snapshots and recordings.
That backfill now runs on every open rather than only on the open that adds the
column, so a store left half migrated is healed rather than left with rows no
owner can see.

Text the change falsified: the audit log records a healed row under the record's
owner, not under a mechanism name, and an already-confirmed record updates its
timestamp and severity but never its reporting sources. The ambiguity refusal is
two sentences instead of a semicolon splice.
A migrated record never carries an empty owner. Where the legacy
reporting_sources holds nothing readable the record gets the synthetic owner
`legacy`, so it stays addressable through source_id like any other record: it
can be read and cleared scoped, and it appears under that name in an ambiguity
refusal. An empty owner left the record unreachable by any call, and because its
evidence rows could then never be assigned to anyone while still looking like
pending work, every open re-entered the migration write transaction and repeated
a warning that nothing would ever resolve. The re-entry check now asks whether
any child row can actually be assigned, which is the backfill's own condition, so
a row nothing can resolve is left alone instead of retried forever.

The empty owner now means exactly one thing, an evidence row not yet assigned to
a record.

Recovery of a value that is neither a JSON array nor a bare word (an object
wrapper, a value behind a byte-order mark) also yields `legacy`. Returning such
text whole minted an owner out of punctuation.

MutedFaultInfo gains a trailing source_id. One entry is one muted record, and two
owners muted on one fault code produced two entries that no reader could tell
apart or address. The owner is filled from the engine's map key, so an entry
cannot name a record other than the one it is filed under.

Dropping a leftover migration scratch table is logged with the table named. The
names faults_new and freeze_frames_new are reserved for the rebuild, and throwing
away debris under them has to be visible to whoever is repairing the database.

Also reworded the prose this branch added that spliced clauses with a semicolon.
Comments and a few messages in the fault manager joined two clauses with a
semicolon. They are split into sentences. Two test comments opened with a
bare label instead of the behaviour they pin, and one test used a
vendor-specific source id. Both now say what they test in plain words.
capture_on_confirm takes the FaultId of the record that confirmed, but
its Doxygen block still described a fault_code parameter. Doxygen warned
twice: once for the unknown fault_code argument and once for the
undocumented id. Name the parameter it actually has.
test_03 still asserted the old aggregate: two nodes reporting one fault
code shared one record that listed both sources. The fault manager now
keeps a record per (fault_code, reporting source), so the test failed
with "'/node_b' not found in ['/node_a']".

Rewrite it to the per-record contract. Two sources reporting one code
give two records, each naming only its own source. Edge counting is
checked per record (a re-report of an active record does not bump its
count), and severity escalates only inside the record whose source sent
the higher severity, so the other source's record keeps its own.
test_06 waited for one SHARED_SENSOR record carrying both forwarded
hardware ids. Each hardware id is now the owner of its own record, so
the wait timed out with only the camera's record in view.

Wait for each hardware id's own record instead and check that the code
has exactly two records, one per hardware id, both CRITICAL. The
publish_until helper takes an optional owner so it can pick one record
of a code that several sources report, and its failure names that owner.
The snapshot cap, the recording cap, the recapture cooldown and the
near-miss series are all kept per fault record, a (fault_code, source)
pair, but the configuration reference, the snapshot tutorial and several
comments still gave the fault code as their unit. The near-miss section
also said the debounce counter was shared by every source of a code,
which is no longer true: each record has its own counter, so an entry's
threshold and counter belong to the same source.

Name the record as the unit everywhere these limits are described, and
say that retrieval of a bag goes through its rows rather than through
the fault code.
Several texts still described a fault code as one record that gathers
every source reporting it:

- the action status bridge said reporting_sources is append-only as the
  reason it never re-attributes a provisional source. The reason now is
  that a swapped source opens a second record and strands the first.
- the roadmap listed multi-source aggregation into a single entry.
- ListRosbags.srv said the entity is matched by prefix against
  reporting_sources, while the fault manager matches the recording's
  owner exactly.
- the gateway README listed reporting sources among what a
  fault_updated event can change. A record's source never changes.
- the package tables described the fault manager as fault aggregation
  and the recapture cooldown as per fault code.

Reword each to the per-record model.
An earlier fault manager opened on a store this release migrated does
not cope with it. It aborts on the first report of a code the store does
not hold yet (NOT NULL constraint failed: faults.owner), and for a code
two sources share it writes one source's report into the other source's
row. Tell operators to keep a copy of faults.db from before the upgrade
when a rollback may be needed, and to restore that copy rather than run
the earlier release on the migrated file.
clang-tidy flagged lines the per-record change added. resolve_target
declared the id it returns const, which stops the return from moving
it (performance-no-automatic-move). Four event waits added with the
per-record tests checked size() >= 1 instead of empty()
(readability-container-size-empty). Drop the const and use empty().
A fault record is the pair (fault_code, owning reporting source), so the four
services that act on one record now receive that owner. GetFault, ClearFault,
GetSnapshots and GetRosbag carry source_id, and the transport's client-side
prefix filter on the GetFault answer is gone: the fault manager resolves the
record, so there is nothing left for the gateway to second-guess. On GetRosbag
the field scopes the fault-code lookup only.

The wire says which record it is describing. The flat fault item gains a
top-level source_id beside reporting_sources, whose single entry it is, and the
SOVD detail gains x-medkit.source_id. A client reading a list or a detail can
address the record it is holding without unpacking an array.

A code addressing several records in one scope is a 409, so error_codes.hpp
gains the vendor code x-medkit-ambiguous-fault for the handlers to use.

ListRosbags.srv documented a prefix match on the reporting source. Both storage
backends match the record owner exactly, so the comment is corrected.
A per-entity route used to read a fault by code and then ask whether the
record it got back happened to be in scope. It now resolves the other way
round: the entity's own reporting sources select the records of that code,
and the route acts on the one it owns, with that owner.

Several in scope is an answer, not a pick. GET and DELETE on
/{entity-path}/faults/{fault_code} return 409 x-medkit-ambiguous-fault with
both owners in params, because acting on whichever record the store listed
first is the failure per-record identity exists to remove and the caller
cannot tell from the response which one it got. None in scope stays 404.

Per-entity DELETE /{entity-path}/faults clears each in-scope record with its
own owner, so two sources of one code no longer leave one record standing.
The global DELETE /faults does the same, and its lock check now resolves an
external app or an external component by its bare id: the map was built from
ROS FQNs alone, which are empty for exactly those entities, so their locks
were silently walked through while the locking page said otherwise.

The fault-trigger engine clears under the app id it reported with, and
entity freeze frames are keyed by (fault_code, owner): a second source
confirming one code used to overwrite the first source's frames and serve
them on the first source's detail page. The startup catch-up dedupes against
a queued confirm by record too, so another owner's standing record still
gets framed.

A rosbag download authorizes on the records attached to the recording rather
than on an unscoped read of each attached code, which with two owners
answered ambiguous and refused an entity its own recording. The descriptor
date lookup is keyed per record for the same reason.

The SSE entity hint resolves a record owned by an external component's bare
id, which matched no app and so carried no hint at all.
send_clear_fault takes the entity it reported under and sends it as the
ClearFault source_id. The plugin already reports under that entity, so this
makes its clear address the record it raised. Without it the clear carried no
source and reached whichever record the store resolved, which on a box running
a second plugin instance against another device is that device's still-active
fault.

FaultProvider::clear_fault passes its entity_id through instead of only
echoing it into the acknowledgement body, and the native-alarm cleared path
names its delivery's entity. AlarmAction::ReportHealed stays a no-op.

The node-map uniqueness guard stays and its comment and the README now name
the real reason: this plugin's own shared FaultTransitionTracker is keyed by
fault_code alone, so two entries emitting one code flap raise/clear every
cycle before either report is sent. The fault manager is not the reason, it
keys a record by code and reporting source.

graph_watchdog says the same thing about its own design: aggregating a
graph-level condition under one owned entity is a choice about what a fault
means, not a workaround for a store that could only hold one record per code.
The fault-detection design page and README separate the tracker's code-only
key from the fault manager's record identity, and the grouping suite's model
note now files a fault record as leaf-owned, which is what it is.
The fault-trigger engine fetched rule values, and enumerated a rule's data
points at create time, through the owning plugin's vendor data route only. A
plugin that serves /data through a DataProvider and registers no such route
answered nothing, so every tick read as "no value right now" and the rule held
its state forever without ever firing or logging a reason.

Provider first, vendor route as the fallback, which is the order the /data
endpoint itself resolves in and the order the freeze-frame capture already
used. The resolution moves to PluginManager::fetch_entity_data_content so the
engine's value fetch, its data-point enumeration and the create-time point
check share one answer and cannot drift. A throwing provider falls through to
the route instead of leaving the engine's evaluation loop.
Two external apps under one component report one fault_code, which is the
shape the whole record model exists for and the one no unit layer runs end to
end: a real fault manager behind a real gateway.

The global list carries one item per owner with its own source_id, each app's
detail route serves its own record, clearing one leaves the other CONFIRMED,
and the component's bulk clear takes both. The component's per-code detail
route answers 409 x-medkit-ambiguous-fault naming both owners, because it owns
two records of that code and the URL names neither. The served OpenAPI
declares the source_id and that 409, and every stream frame names the record it
describes.

The manifest fixture is two external apps on one component, so both records
land in one fault scope without needing a plugin.
The correlation view listed a muted entry by fault_code alone, which addresses
as many records as there are sources reporting that code. A client reading
x-medkit.muted_faults could not tell which record the correlation engine muted,
nor address it to inspect or clear it.

The entry now carries the owning source the fault manager reports alongside it,
so a muted entry names one record the same way every other fault item does.
Six defects, all reachable from a client, all the same shape: a fault code
travelling somewhere a record was meant.

The plugin clear path carried the addressed entity where the owner belonged.
A component owns the records its hosted apps reported, so DELETE on a
component's fault sent the component id as source_id, the fault manager found
no record under it, and the route answered 2xx with the record still
CONFIRMED. FaultProvider::clear_fault now takes the owner the gateway
resolved (defaulted empty for a provider that cannot scope), the handler hands
it scoped->owner, the bulk clear hands each listed item its own source_id, and
the OPC UA plugin sends that owner. Its dispatch also reads the reply now: a
refused clear used to leave no trace anywhere.

A DataProvider's list envelope carries connected and timestamp, the shape the
plugin's own data route already served. Reading the provider first without it
meant the trigger fetcher's link-down guard could never fire for an OPC UA
app, so a threshold rule evaluated on the frozen last-known values the poller
keeps for the whole outage.

Muted records are addressable again. Resolving out of a listing that excludes
them 404'd a record on the per-entity GET, DELETE and bag download that was
reachable before, because muting is the correlation engine hiding a symptom
from a listing, not an unaddressing.

The detail's record owner is x-medkit.owner. It collided with a fault list's
x-medkit.source_id, which means something else entirely: the addressed
entity's namespace path, empty for an external app. One key, two meanings,
inside one API.

A bag URL carrying a bare fault code refuses to pick an owner. It used to
serve the lowest-sorting owner's recording under a URL that never said whose
bytes those were, so it now answers the same 409 the fault routes do. The
recording-id form addresses the bag directly and is unaffected.

The per-entity DELETE declares its ambiguity 409 explicitly rather than
relying on the one its lock marker contributes.

Documentation: the trigger-code collision is engine-wide and not per app, the
error envelope key is parameters and not params, and the two owner-ish keys
are named apart. A test fixture now gives one entity both a provider and a
vendor route, which is the only shape in which the provider-first order is
observable at all.
Every per-entity fault list leaves muted records out, but the per-code
routes resolved a code over muted records as well. A component whose list
showed one record of a code answered DELETE with 409 because another
owner's record of that code was muted, and the 409 told the client to
read the owners from a list that could not show one of them.

A code now resolves over the records the entity's list shows. Only when
it shows no record of the code do the muted records answer, so a muted
symptom stays addressable without making a shown record ambiguous. The
new faults::addressable_records applies the rule from the listing's
muted_faults entries, matched on code and owner, and the detail route,
the native clear, the plugin clear and the recording download by fault
code all resolve through it.

The 409 now points the client at parameters.owners and at the owning
app's route instead of at the entity's fault list.
The per-code DELETE on an entity whose faults a plugin serves reads the
fault manager first, to learn which record the code names. When that
read failed the owner stayed empty and the provider was called anyway,
so the route answered 2xx while the record it meant stayed untouched,
and the comment above it described the empty owner as "no record".

A failed read now answers 503 exactly as the native path does, and the
provider is never called with an owner the gateway did not resolve. The
owner is empty only when the fault manager answered and holds no record
of the code at all, which is the plugin-internal case the provider
decides for itself.

The REST reference lists the 409 on the per-code routes, the plugin's
200 on the DELETE, and the 503 for plugin-served entities.
Clearing one fault record needs its owner, and the owner reached the
provider by widening clear_fault(entity_id, fault_code) into a
three-argument pure virtual while the plugin API version stayed at 7.
A provider that overrides the two-argument clear_fault no longer
compiled, and a provider binary built against the old headers still
passed the loader's equality check.

clear_fault(entity_id, fault_code) is back exactly as it was. The new
virtual clear_fault_record(entity_id, fault_code, owner) is what the
gateway calls for the per-code and the bulk DELETE, and its default
implementation calls clear_fault(entity_id, fault_code), so provider
source written against v7 compiles unchanged and keeps receiving the
clear. The OPC UA plugin overrides clear_fault_record so its clear
names the owner the gateway resolved.

PLUGIN_API_VERSION is 8: the new virtual changes the FaultProvider
vtable, so a pre-built v7 plugin is rejected and has to be rebuilt.
The plugin tutorial, the OPC UA design notes and README, and the graph
watchdog design note describe the new contract and version.
A rosbag download URL carrying a fault code resolves that code to one
record the way the fault routes do, and answers 409
x-medkit-ambiguous-fault when several records in the entity's scope
carry it. None of the six download routes declared that 409, and the
REST reference listed 200, 206, 404 and 416 only.

The 409 is now declared on every download route and documented. Its
details also sent the client to x-medkit.recording_id on the fault
listing, which the listing does not carry. They now name where the
recording id really is: each descriptor id of the entity's rosbags
listing, and a fault detail's environment_data.snapshots[].bulk_data_uri.
The owner-identity launch ran with locking on, the shipped default. The
per-code DELETE is lock-guarded, and that marker adds a 409 of its own
to the document, so the assertion that the DELETE declares its
ambiguous-fault 409 stayed green with that 409 removed from the route.

The launch now runs with locking off, which drops the lock contract
from the document, and the test first grounds that: the root reports
capabilities.locking false and the DELETE carries no lock marker. A
409 left on the DELETE is then the route's own, and removing it turns
the test red.
The REST reference and the gateway README said a muted record is still
addressable "on all of these routes" right after describing the bulk
DELETE /{entity-path}/faults. The bulk DELETE lists the entity's
records without muted ones, on the native path and through the plugin
context alike, and it always has.

That behaviour stays, and the text now states it: the bulk DELETE
clears the records the entity's fault list shows, and a muted record is
cleared by its own per-code DELETE. A test pins both halves: the bulk
clear leaves a muted record CONFIRMED while it clears the shown one,
and the muted record's per-code DELETE then clears it.
Creating a fault trigger rule whose fault_code another rule already
holds is refused with 409, and the message said fault codes are global
to the fault store. They are not: the fault manager keys a record by
code and reporting source, so rules on two apps would own two separate
records. The comment above the check already said the refusal is the
engine's own rule, and the message now says the same: the trigger
engine keeps one rule per fault code across every app, so a code names
exactly one rule's condition.
The ClearFault reply callback captured `this` to log a refused clear.
The reply is handled on an executor thread whenever it arrives, and
shutdown() drains nothing, so a reply handled while the plugin was
being destroyed ran against freed memory. The callback now holds
nothing of the plugin: it logs through a copy of the log sink taken
when the request is sent, and carries the owner and the code by value.

GatewayPlugin gains a protected log_sink() that hands out that copy,
and set_logger() becomes protected so a plugin hosted without a
PluginManager can wire a sink of its own. Both are non-virtual.

The refused-clear test had no log sink, so it stayed green with the
reply read removed. It now wires one and asserts the warning names the
code and the owner. A second test swaps the plugin's sink while a reply
is held back and checks the reply still logs to the sink the plugin had
when it sent the clear, with a second clear as the positive control.

Also drops a useless int64_t cast in list_data, and three test comments
lose a leading label that named nothing a reader can look up.
Comments, doc text and a few log and error strings introduced with the
per-record fault identity joined two clauses with a semicolon. They
now use a full stop or a comma, which reads the same and keeps the
prose style of the surrounding text. The duplicate-code error of the
OPC UA node map also ends with a sentence instead of a dash.

A snapshot-capture test fixture app gets a neutral id.
Formatting only: line breaks and include order that clang-format expects in the files the per-record resolution, the plugin clear and the trigger message changes touched.
The OPC UA plugin's component status route checked only that the
component exists and then answered with the plugin's own session
snapshot under whatever id the path named. In a gateway that loads
several plugins, a status request for another plugin's component
therefore answered 200 with this plugin's endpoint, mode and link
state, as if they described that component.

The handler now compares the requested id with the component the
plugin introspects, read under the node map lock it already holds, and
answers 404 resource-not-found for any other component. That is the
answer the data route gives an entity with no mapped points. An id
that names no entity at all still gets the entity validator's own 404,
and the plugin's own component is served as before.

The single data point route and the write route already look the
entity up in the node map and answer 404 for an entity the plugin does
not map, so they are unchanged.

The test stubs for the plugin request and response now carry path
parameters and record what a handler sent, so a route handler can be
driven directly. The new test asks for the plugin's own component
first, as the positive control, then for a second component.
A per-entity route resolves a fault code in the URL to the records of
that code in the entity's scope. The candidates were every record of
the code the store held, muted ones only when nothing else matched,
and CLEARED and HEALED records counted alongside active ones. Once two
sources had reported a code and one of them cleared its record, the
component's per-code GET, DELETE and bag URL answered 409 for as long
as the cleared record was kept, while the component's fault list showed
a single record.

Resolve in three tiers and let the first tier holding any record
decide: the records the entity's default fault list shows (PREFAILED
and CONFIRMED, not muted), then the muted records of those statuses,
then everything the list shows only on request (CLEARED, HEALED and
PREPASSED). One record in the deciding tier is acted on, several still
answer 409 naming their owners. A cleared record alone stays
addressable.

The REST reference, the gateway README, the OpenAPI note and the
Postman descriptions state the tiers.
A recording downloaded by its id was authorized when any record of an
attached fault code was in the entity's scope. Two sources reporting one
code are two records, each with its own recordings, so either app could
download the other's recording of that code, and still could after its
own record had been cleared.

A recording belongs to the (fault code, owner) records it is attached
to. GetRosbag names the attached codes but not their owners, so the
download now takes the owners of the entity's own records of those
codes and asks ListRosbags, which answers exactly the rows one owner
holds, whether one of them holds the served recording. The owners are
the entity's fault scope, the scope its fault list uses. A recording
matches a row by its id,
or by its bag path when either side predates recording ids.

Tests cover both apps of one code downloading only their own recording,
the hosting component downloading both, and the app that cleared its
record still refused the other's recording, against a stub fault manager
and over HTTP against the real one. The REST reference and the gateway
README say which recordings an entity serves.
Under replay-buffer pressure the fault stream coalesces an update that a
newer event of the same fault supersedes, and evicts superseded
transitions before anything else. "The same fault" was keyed on the
fault code alone. Two sources reporting one code are two records, so a
newer event of one owner's record coalesced the other owner's last
update away without counting it, and a lagging client never learned
that record's current state.

Key the supersede check on the record: fault code and owner, from the
same peer. The eviction notes in the handler, the stream entity hint in
the frame DTO, the gateway README and the REST reference now speak of
the record's single owner instead of several co-reporters.
clang-tidy flagged the ClearFault reply callback twice: the response
was copied out of the future, and the future itself is taken by value.
Bind the response by const reference.

The future has to stay a by-value parameter. rclcpp's async_send_request
only accepts a callback whose parameter type is exactly SharedFuture,
and a const reference is a different type to that check, so the
callback cannot take it by reference. The check is suppressed on that
one line with the reason next to it. The callback moves into a named
lambda so the suppression sits on the line clang-tidy reports.
Three comments still reasoned from a fault manager that keys a record
by fault code alone: the poller's guard against a stale condition
instance clearing a live one, the node map test for a code shared
across entities, and the plugin's fault list fallback for a missing
source_id. A scope test also called a two-source record co-reported.

Say what holds now. The poller guard is still needed because both
condition instances report under the one entity that owns the code, so
they address one record. The node map rejects a shared code because the
poller's own transition tracker keys by code. A record carries its
owner as source_id, and a record with two sources can only come from a
peer or an older store.
The recording ownership text said an entity downloads exactly the
recordings its rosbags listing carries. The listing asks for the rows of
each of the entity's own sources by exact match, while the download
accepts the owner of any record in the entity's fault scope, and that
scope also admits an owner below one of those sources. Say that the
download uses the fault list's scope, and keep the exact statement to
what holds: a recording downloads when a listing under one of those
owners carries it.
The fake fault manager services the OPC UA plugin tests start took
their request, response and header shared pointers by value
(performance-unnecessary-value-param). rclcpp accepts service callbacks
that take them by const reference, so take them that way, the deferred
ClearFault callback included. The recording listing test built each
download path with two string concatenations inside a loop
(performance-inefficient-string-concatenation). Append the id instead.
// newer state of the other would delete a live fault from a client's view.
auto supersede_key = [](const QueuedEvent & queued) {
return queued.peer + '\0' + queued.event.fault.fault_code;
const auto & sources = queued.event.fault.reporting_sources;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_peer_eventreads onlyfault_codeback out of a relayed payload, neverreporting_sources, so for every relayed entry owneris empty here and the key collapses to (peer, code): a peer's two records of one code still coalesce into one, and a lagging client loses one owner's last update uncounted, which is exactly what this commit removes for local events. Readfault.reporting_sources(orsource_id) back into queued.event.faultinon_peer_eventand cover it with a relayed variant ofAnotherOwnersEventsDoNotSupersedeARecordsLastUpdate`.

// shows first, then the muted ones, then the cleared or healed ones, so
// neither a muted record nor one a source cleared turns the record a client
// read off the list into an ambiguous address.
auto result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every per-code GET and DELETE now pulls the whole store (all statuses, muted included) through ListFaults before the one GetFault it used to make, and the plugin clear branch and the recording download repeat the same listing, so with cleared records retained indefinitely each request costs O(history) on routes a stream consumer hits once per event. Call get_fault_with_env(fault_code, "") first, which the manager answers only when the code names exactly one record, scope-check that record, and take the full listing only when that call does not succeed: the tiers only matter with two or more records, so nothing observable changes.

auto held = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true,
/*include_muted=*/true, /*include_clusters=*/false);
if (!held.success) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plugin DELETE now returns 503 when ListFaults fails, but plugin GET still falls through to the provider in that case. On a plugin-only box (UDS DTCs, no fault_manager node) read works and clear of the same DTC 503s. Make both paths agree: fall through with an empty owner here, or 503 in get_fault too.

make_error(409, ERR_AMBIGUOUS_FAULT, "Fault code addresses several records in this entity",
json{{"details",
"Several sources this entity owns report this fault code, and each is its own record. "
"parameters.owners names them. Address one through the route of the app that owns it, "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hint points at /apps/{app_id}/faults/{fault_code}, but owners are reporting sources: a node FQN for a ROS app, and a bare component id for an external component, which has no app route. If an external component and one of its apps report the same code, the component's own record cannot be read or cleared by code at all. Return each owner's entity route (build_source_entity_map already resolves it), or let the caller pick the owner.

auto held = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true, /*include_muted=*/true,
/*include_clusters=*/false);
const json listing = held.success ? held.data : json::object();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failed list_faults becomes an empty listing, so requested_records and candidate_owners are empty and a bag this entity owns answers 404 "not found for this entity". Return classify_fault_failure (503) on !held.success, like the fault routes, and declare it on the download routes.

// shows first, then the muted ones, then the cleared or healed ones, so
// neither a muted record nor one a source cleared turns the record a client
// read off the list into an ambiguous address.
auto result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every per-code GET/DELETE now lists the whole store (all entities, cleared and healed history, muted) and then does a GetFault, both under the transport's executor mutex. Cost grows with retained history. List per scope source (list_faults(source)) or add a fault_code filter to ListFaults. Same full read in the plugin clear (997), download (494) and descriptor listing (376).

}
}
if (lock_mgr) {
auto it = source_to_entity.find(owner);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact find(owner) misses path-child sources, which the per-entity scope gives to the parent app. A record owned by /ns/app/child is cleared through a lock on /ns/app, contrary to the new locking.rst line. Match with the same rule as source_matches_scope (exact or strict path child).

{
auto faults_result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true,
/*include_muted=*/false, /*include_clusters=*/false);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include_muted=false drops muted records from faults, so a muted record's legacy rows (created_at_ns == 0) get dated 0 although download serves them. Use include_muted=true, as download does.

/// The reporting source that owns this record. Together with `fault_code` it
/// addresses the record on the per-record routes, and it is the single entry
/// of `reporting_sources`.
std::optional<std::string> source_id;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A list item now has top-level source_id = owner and x-medkit.source_id = namespace path, while the detail calls the owner x-medkit.owner to avoid exactly that clash. Use owner here too.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants