Skip to content

fix(opcua): reconnect, alarm and companion specification fixes - #783

Open
mfaferek93 wants to merge 12 commits into
mainfrom
fix/opcua-reconnect-alarms-companion-specs
Open

mfaferek93 wants to merge 12 commits into
mainfrom
fix/opcua-reconnect-alarms-companion-specs

Conversation

@mfaferek93

Copy link
Copy Markdown
Collaborator

Summary

Twelve changes to the OPC UA plugin, one commit each, in the order below. They were open as six separate pull requests (#777, #778, #779, #780, #781, #782). Together they make the plugin survive a dropped secured link, say why it is down, address nodes by companion specification browse paths, gate detection rules on machine state, and keep native alarms correct with several alarm sources.


1. Empty BrowsePath in the ConditionId clause (was #777)

What. The event filter sent with every Alarms & Conditions subscription has a ConditionId select clause with no BrowsePath (ConditionType, AttributeId=NodeId). The plugin left browsePath as a NULL pointer, which open62541 encodes as a null array (length -1). The clause now carries an empty array (length 0): make_event_filter sets browsePath to UA_EMPTY_ARRAY_SENTINEL when the path is empty. It moves to a detail namespace so a test can call it.

Why. A Siemens S7-1500 answers a null BrowsePath with a null ConditionId. The plugin drops every event with a null ConditionId as a non-condition event, so no native Program_Alarm ever became a fault:

OPC-UA non-condition event on configured alarm source 'ns=0;i=2253' (fault_code 'PLC_ALARM') carries a null ConditionId and was dropped

How tested.

  • New EventFilterTest.ConditionIdClauseSendsEmptyNotNullBrowsePath: encodes the clause, decodes it and checks it decodes as an empty array, not a null one. It fails on main (browsePath decodes as NULL) and passes with the fix.
  • On a CPU 1505SP F (FW V21.9): the plugin connects, the event monitored item with the new filter is accepted (server diagnostics show the item and the ConditionRefresh notifications), and polling is unchanged.

2. Lock the shared log check in the announce-before-sweep test (was #777)

What. DiscoverEndpoint.TheScanIsAnnouncedBeforeTheSweepRuns gets a mutex around the check and the assignment of a shared string.

Why. The scan lambda runs on the sweep's worker threads. Two first probes could both see the string empty and assign it at once, which ThreadSanitizer reports as a data race. It only shows when two probes overlap, so it is rare.

How tested. A short sleep between the check and the assignment makes the race show on every TSan run. With the mutex and the same sleep, 10 of 10 runs are clean.

3. A secured client's config lives as long as the client (was #778)

What. The UA_ClientConfig a secured client is built from lives on the heap in OpcuaClient::Impl (declared before client, so it is destroyed after it). It is replaced only once the client built from the next one is in place. After the move it holds a default config, so the policy context logs through a valid stdout logger.

Why. A gateway connected to a secured OPC UA server (Basic256Sha256, SignAndEncrypt) crashed with SIGSEGV when the link dropped:

#0  0x005b205d32353138 in ?? ()
#1  UA_LOG_INFO.constprop.0 ()
#2  UA_ChannelModule_Delete_Context ()
#3  UA_SecureChannel_close ()
#4  closeSecureChannel ()
#5  receiveResponse ()
#6  UA_Client_run_iterate ()
...
#9  ros2_medkit_gateway::OpcuaPoller::event_pump_loop()

In open62541 1.3 (bundled by open62541pp v0.16), each security policy context stores a pointer to the logger of the UA_ClientConfig it was built in. apply_security_config built that config as a local opcua::ClientConfig and moved it into the client. The existing 1.3 workaround re-points securityPolicies[i].logger, but the policy context keeps its own copy of the pointer (ua_openssl_basic256sha256.c, cc->policyContext->logger). Once connect() returned, that pointer led into a dead stack frame. Closing a secured channel logs through it, so the crash came whenever that stack memory had been reused before the next link drop.

How tested.

  • New SecuredOpcuaClientReconnect.ChannelClosesAfterConnectReturned: secured connect to test_alarm_server --secure behind the link proxy, overwrite the dead stack, cut the link, pump until the channel closes, reconnect, disconnect. Without the fix it dies deterministically with SIGBUS at 0xa5a5a5a5a5a5a5a5 in UA_ChannelModule_Delete_Context, the stack seen in the field. With the fix it passes.
  • The crash came twice in a few minutes on an aarch64 gateway reading a secured server through a link that went down and up. Same backtrace from the core dump.

4. x-plc-status says why the link to the server is down (was #779)

What. GET /components/{id}/x-plc-status gains last_error: the OPC UA status name of the last failure of the link.

  • a connect the server or the client refused: BadCertificateUntrusted (the server's certificate is not in the trust list), the status of the server's ERR message such as BadSecurityChecksFailed (the server does not trust the client's certificate), BadUserAccessDenied (a refused user), BadTimeout / BadConnectionClosed (nothing answers)
  • BadConfigurationError when the client's own certificate, key or security settings could not be applied
  • a session that dropped, or a value read the server refused

It is cleared by a connect that succeeds. It has its own lock, so the route never waits behind a connect in progress.

Why. A client of the gateway saw connected: false and nothing else. A missing trust entry on either side, a wrong user and a server that is off all looked the same. open62541 1.3 ends a refused channel with BadConnectionClosed (ua_client.c receiveResponse) and only logs why. connectStatus holds the same code. So the client puts two taps on its own config: one in front of the certificate check of the server's certificate, and one in front of the logger that matches the exact format strings of processERRResponse. Both forward to the original plugin.

How tested.

  • test_opcua_client: a refused port names a Bad... status that is not a certificate one, a contradictory security config gives BadConfigurationError, a new client has none.
  • test_opcua_secured (real open62541 server, Basic256Sha256 SignAndEncrypt): wrong password gives connected == false, last_error == BadUserAccessDenied. Server certificate not in the gateway's trust list gives connected == false, last_error == BadCertificateUntrusted and no value served (without the taps this reads BadConnectionClosed). A second server that does not trust the gateway's certificate gives last_error == BadSecurityChecksFailed.

5. A refused session is not a connection, and a missing token policy says so (was #779)

What. Only an activated session counts as connected. A server that offers no token policy for the configured user is reported as BadIdentityTokenInvalid (for example anonymous on a server that wants a user) instead of open62541's BadInternalError.

Why. A channel left open by a refused ActivateSession read as connected through open62541pp's isConnected(). The fast path of connect() returned true, x-plc-status said connected: true and every read failed. With this check the client rebuilds on every retry of a refused session, which crashed before change 3 (SIGBUS in UA_ChannelModule_Aes128Sha256RsaOaep_Delete_Context, logging through a policy context's logger pointer to a dead stack frame). That is why change 3 comes first.

How tested.

  • test_opcua_secured: anonymous gateway, server without an anonymous policy gives connected == false, last_error == BadIdentityTokenInvalid. Without the session check this reads connected: true with every read failing.
  • Debug build of main plus the session check: SIGBUS on the third connect of a refused session. With change 3: none.

6. Node map entries addressed by browse path (was #780)

What. A node map entry can give a browse_path instead of a node_id:

namespaces:
  di: "http://opcfoundation.org/UA/DI/"
  rob: "http://opcfoundation.org/UA/Robotics/"
nodes:
  - browse_path: "di:DeviceSet/*/rob:SafetyStates/*/di:ParameterSet/rob:EmergencyStop"
    entity_id: robot_safety
    data_name: emergency_stop
    data_type: bool
    alarm: {fault_code: ROBOT_EMERGENCY_STOP, severity: ERROR, threshold: 0.5, above_threshold: true}

Why. Companion specifications (OPC 40010 Robotics, Machinery, PackML, ...) fix the BrowseNames and the namespace URI of their nodes. They do not fix NodeIds, and the namespace index is the server's choice, which can change when it restarts. A map written against one server's NodeIds only fits that server. A browse path written against the specification fits every server that implements it.

How.

  • Steps from the Objects folder: <alias>:<BrowseName> with aliases from a new top-level namespaces: map, <index>:<BrowseName>, or * for any child (an instance the vendor names). The last step must be a Variable.
  • An entry with a browse path waits outside entries() until a session resolves it, so the poller never subscribes a null NodeId. It is resolved after connect, before auto_browse and before the subscribe, and again from on_connected on every new session: a path that was missing is retried, and a node that moved (namespaces renumbered) is re-keyed, publisher included.
  • Under *, children are tried in browse order with backtracking, first full match wins, capped at 2000 children per path.
  • Explicit wins: a resolved entry replaces an auto_browse entry for the same node. Fault-code uniqueness covers both kinds of entry at load.
  • Read-only: Browse and a NamespaceArray read only.

How tested.

  • New test_browse_path: parsing and its refusals, wildcard walk with backtracking, namespace matched by URI and not by index, last step must be a Variable, node budget, waiting entries, replacing an auto-browsed entry, following a node that moved on the next session, fault-code uniqueness, one address per entry.
  • End to end against an asyncua server built on the DI 1.04 + Robotics 1.01.2 nodesets: entries resolve after a failed first connect, the server restarting with two more namespaces in front is followed to the new NodeIds, and a link lost and restored keeps the faults. A gateway without this change skips the same entries.

Not in this change.

  • auto_alarms folds the SourceNode NodeId string, index included, into the derived fault code, so the same condition gets another code after the namespaces are renumbered.
  • auto_browse entries from the previous session stay after a renumbering until the gateway restarts.

7. State registers with several healthy values, and gates (was #781)

What.

  • fault_enum.ok_values: more healthy values beside ok_value.
  • inhibit_when / only_when on any detection entry, gated on another polled point of the map:
    • inhibit_when: no fault is raised while the gate point reads one of values (changeover, setup, maintenance mode). A fault raised before keeps its state until its own value clears it.
    • only_when: the rule is evaluated only while the gate matches, otherwise its faults clear (a stop reason that stays latched after the machine runs again).
    • The load fails when the gate names a point that is not in nodes:, names the entry itself, or values is not a non-empty integer list. An unread gate point gates nothing.
  • A gate can sit on a browse_path entry (change 6). The gate point itself must be a node_id entry, because a browse path entry has no NodeId until a session resolves it. The load checks the gates of both kinds of entry.

Why. A state machine register (ISA-TR88 state numbers, Machinery item state) has many healthy states. Without the list each of them raises the catch-all. Many machine faults only mean something in some modes of the machine.

How tested.

  • test_fault_detection: OkValuesAreHealthyStates.
  • test_node_map: parsing of ok_values and both gates, rejection of a gate on a missing point, on itself, and malformed lists.
  • test_browse_path: AGateOnABrowsePathEntryJoinsWithIt (the gate survives the load and the resolve), RejectsAGateWhosePointIsMissingOrAddressedByBrowsePath.
  • New integration test test_opcua_state_gates (real gateway, fault manager and fixture server): healthy states, aborted in production raises, a raised fault survives an inhibiting mode, aborted during an inhibiting mode is not raised and is raised once the mode ends, catch-all, only_when ignores a latched reason while running and raises it while stopped, planned reason raises nothing. Against the plugin without this change it fails 7 of its expectations.

8. Array values are skipped instead of terminating the gateway (was #781)

What. A non-scalar value is now a bad read, so auto_browse skips the node and polling or a subscription drops it.

Why. A read of an array value terminated the gateway. variant_to_value handles scalars only and getScalarCopy throws BadVariantAccess on an array of a matching type, which no read path caught. auto_browse against any server with array variables (seen on the umati sample server) disabled the plugin in set_context or terminated the process on a later walk.

How tested. The fixture server gets three Int32 registers and a read-only Int32 array (Recipe.Steps). Without the fix, test_opcua_read_only and test_opcua_state_gates fail with the plugin disabled (Plugin 'opcua' threw during set_context(): Variant is not a scalar). With it both pass.

9. ConditionRefresh reconciled per event monitored item (was #782)

What. The replayed set and the reconcile are kept per alarm source, and a RefreshEnd without its RefreshStart reconciles nothing.

Why. Part 9 brackets the refresh replay on every event item of the subscription, and each item carries only its own notifier's conditions. With two event_alarms sources, the second item's RefreshEnd reconciled every tracked condition against its own burst and cleared the first source's live alarms after every reconnect.

How tested. test_opcua_poller: ConditionRefreshTest.EachSourceReconcilesOnlyItsOwnConditions (two sources, refresh bracketed on both, then one source alone, then a stray End). With the old reconcile it fails ({ "B_ALARM", "A_ALARM" } cleared, want { "A_ALARM" }).

10. An array event field no longer drops the notification (was #782)

What. Every scalar read in the event path and in the read-based condition replay checks isScalar() first.

Why. Change 8 makes an array value a bad read in the read path. The event path had the same isType<T>() + getScalarCopy pattern: an array Severity, Message, ConditionName or associated value threw out of on_event, and the notification was dropped. Same in the read-based condition replay.

How tested. test_opcua_poller: EventFieldsTest.AnArrayFieldDoesNotDropTheEvent. It fails without the event-path guard.

11. Report and clear of one fault code kept in order (was #782)

What. Calls for one fault code go out one at a time, each after the previous was answered (FaultCodeLanes). Other codes do not wait, and an unanswered call holds its code for at most 5 s.

Why. ReportFault and ClearFault are separate services, so an alarm that came and went inside one publishing interval could reach fault_manager clear first and stay raised for good (seen: Fault cleared: X followed by New fault reported: X).

How tested.

  • New test_fault_code_lanes (no ROS): order kept for one code, other codes do not wait, several calls in a row, stale call released and its late answer ignored, synchronous answers.
  • End to end with a real gateway and fault_manager against an asyncua server with two machine objects as notifiers: before changes 9 and 11 a reconnect cleared live alarms, and a raise followed by a clear inside one publishing interval left the fault raised (about 1 run in 3). After: 11 consecutive full runs passed (8 with link reset, 3 with silent link loss).

12. A read-only guard alone no longer turns auto_browse on (was #782)

What. Without enabled, only a walk knob (root_nodes, max_depth, ...) turns auto_browse on. match_message is accepted by the event_alarms mapping key check.

Why. The object form defaulted enabled to true, so auto_browse.infer_writable: false set beside a curated node map turned the walk on. match_message was parsed but logged as an unknown key and "ignored".

How tested. test_opcua_plugin: InferWritableAloneKeepsTheWalkOff, WalkKnobWithoutEnabledTurnsTheWalkOn.


Testing

All runs on the combined branch, in ros:jazzy-ros-base (arm64), read-only build unless noted.

  • Every commit builds: ros2_medkit_fault_detection and ros2_medkit_opcua built at each of the 12 commits, tests included.
  • Release (--packages-up-to ros2_medkit_opcua ros2_medkit_fault_manager, ROS2_MEDKIT_OPCUA_SECURE_REQUIRE=1), colcon test with linters:
    • ros2_medkit_opcua: ctest 31/31 (gtest, the reconnect suite, all launch integration tests, clang_format, copyright, cppcheck, flake8, lint_cmake, pep257, xmllint). test_alarm_server_smoke runs with asyncua installed.
    • ros2_medkit_fault_detection: ctest 7/7.
    • colcon test-result: 639 tests, 0 errors, 0 failures, 39 skipped (cppcheck cases, which ament skips on this distro).
    • All 31 new gtest cases from the six changes and the two launch tests (test_opcua_secured with its four last_error cases, test_opcua_state_gates) pass.
  • ASan + UBSan (-DCMAKE_BUILD_TYPE=RelWithDebInfo -DSANITIZER=asan,ubsan, timeouts x3, MEDKIT_TEST_TIME_SCALE=3, the CI ASAN_OPTIONS / UBSAN_OPTIONS): ros2_medkit_opcua ctest 24/24, ros2_medkit_fault_detection 2/2. 446 + 17 test cases, 0 failures, no sanitizer report.
  • TSan (-DSANITIZER=tsan, ctest -j1, the repository suppressions, timeouts x3): ros2_medkit_opcua ctest 24/24, ros2_medkit_fault_detection 2/2. 446 + 17 test cases, 0 failures, no ThreadSanitizer warning.
  • Docker alarm suite (run_alarm_tests.sh): read-only 25 checks and write-capable 23 checks, all scenarios passed (lifecycle, shelving, disabled alarm, status word, fault enum, reconnect after server restart).
  • OpenPLC suite (run_integration_tests.sh, read-only image): 29/29.
  • clang-format 18: clean on all changed files.
  • clang-tidy 18 (run-clang-tidy, -header-filter='.*ros2_medkit.*') on the 21 changed C++ files: no finding on the lines this branch adds or changes.

…ll one

The ConditionId select clause has no BrowsePath. It went out as a null
array (length -1), and a Siemens S7-1500 answers that with a null
ConditionId, so every native alarm was dropped as a non-condition event.
It now goes out as an empty array (length 0).
The scan lambda runs on the sweep's worker threads. Two first probes could
both see the string empty and assign it at once, which ThreadSanitizer
reports as a race. A mutex around the check and the assignment fixes it.
open62541 1.3 security policy contexts keep a pointer to the logger of the
config they were built in. That config was a local of the connect call, so
the log line on closing a secured channel read a dead stack frame, and a
dropped link crashed the gateway (SIGSEGV in UA_ChannelModule_Delete_Context).
x-plc-status gains last_error, the OPC UA status name of the last failure:
a refused server certificate, an ERR from a server that refuses the client's
certificate, a refused user, a timeout. open62541 1.3 ends a refused channel
with BadConnectionClosed and only logs why, so the reason is taken from the
client's certificate check and from the server's ERR message.
…n policy says so

A channel left open by a refused ActivateSession read as connected through
open62541pp's isConnected(), so x-plc-status said connected and every read
failed. Only an activated session counts now. A server that offers no token
policy for the configured user is reported as BadIdentityTokenInvalid instead
of open62541's BadInternalError. Built on the secured config lifetime fix.
A companion specification fixes BrowseNames and namespace URIs, not NodeIds or
namespace indices. A node map entry may now give a browse_path from the Objects
folder (namespace aliases, * for a vendor-named instance) instead of a node_id.
It is resolved on every new session, so a server that renumbers its namespaces
after a restart is followed to the new NodeIds.
fault_enum takes ok_values, so a state machine register with many healthy
states does not raise its catch-all. A detection entry can be gated on
another polled point: inhibit_when keeps a fault from being raised while
that point reads one of the listed values, only_when evaluates the rule
only then.
The value conversion only handles scalars, and getScalarCopy throws
BadVariantAccess on an array of a matching type, which no read path
caught. auto_browse on a server with array variables (recipe tables,
alarm lists) took the gateway down. A non-scalar read is now not good,
so auto_browse skips the node and polling or a subscription drops it.
Part 9 brackets the refresh replay on every event item of the subscription,
and each item only carries its own notifier's conditions. With two alarm
sources the second RefreshEnd reconciled every tracked condition against its
own burst and cleared the first source's live alarms. The replayed set and
the reconcile are now kept per source, and an End without a Start is ignored.
isType<T>() is true for an array of T, so getScalarCopy threw
BadVariantAccess when an event field (Severity, Message, ConditionName, an
associated value) or a condition property read in the read-based replay was
an array: the notification was dropped, or the throw left the poll thread.
Every scalar read in those paths now checks isScalar() first.
ReportFault and ClearFault are separate services, so an alarm that came and
went inside one publishing interval could reach fault_manager clear first and
stay raised for good. Calls for one code now go out one at a time, each after
the previous was answered (an unanswered call holds its code for at most 5 s).
plugins.opcua.auto_browse.infer_writable: false, set on its own beside a
curated node map, enabled the walk because "enabled" defaulted to true for
any object. Without "enabled" only a walk knob turns it on now. Also accept
match_message in the event_alarms mapping key check: it was parsed but
reported as an unknown key.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved NodeId remapping collisions can leave browse-path entries bound to stale IDs and reading the wrong variables after namespace renumbering.

5 open issues
What changed in this PR

This PR strengthens the OPC UA plugin’s connection recovery and alarm handling, while adding companion-specification addressing and machine-state-aware fault detection.

Changes:

  • Preserve secured-client configuration lifetime, report connection failures through last_error, and reject non-scalar values safely.
  • Add namespace-aware browse paths, multiple healthy enum values, and state gates.
  • Reconcile native alarms per source, sequence fault reports and clears, and expand regression coverage.

Paths below are relative to src/ros2_medkit_plugins/ros2_medkit_opcua/ unless otherwise shown.

File Description
test/​test_opcua_poller.cpp Tests per-source refresh and array event fields.
test/​test_opcua_plugin.cpp Tests auto-browse defaults and synchronizes shared test state.
test/​test_opcua_client.cpp Tests error reporting and empty ConditionId browse paths.
test/​test_opcua_client_reconnect.cpp Adds secured-channel reconnect regression coverage.
test/​test_node_map.cpp Tests healthy-state lists and gate validation.
test/​test_fault_code_lanes.cpp Tests per-code dispatch ordering and stale responses.
test/​test_browse_path.cpp Tests path parsing, resolution, precedence, and remapping.
test/​integration/​test_opcua_state_gates.test.py Exercises gated faults through a running gateway.
test/​integration/​test_opcua_secured.test.py Verifies authentication and certificate failure reporting.
test/​fixtures/​test_alarm_server/​test_alarm_server.cpp Adds state registers and an array fixture.
src/​opcua_poller.cpp Applies state gates and per-source alarm reconciliation.
src/​opcua_plugin.cpp Integrates path resolution, status errors, and ordered dispatch.
src/​opcua_client.cpp Fixes secured-client lifetime, session checks, and scalar handling.
src/​node_map.cpp Parses and resolves browse paths and state gates.
src/​address_space_browser.cpp Adds bounded, namespace-aware path traversal.
README.md Documents status errors and new configuration options.
include/​ros2_medkit_opcua/​opcua_poller.hpp Defines per-source refresh tracking and test access.
include/​ros2_medkit_opcua/​opcua_plugin.hpp Declares path resolution and fault-dispatch lanes.
include/​ros2_medkit_opcua/​opcua_client.hpp Exposes error status and namespace lookup.
include/​ros2_medkit_opcua/​node_map.hpp Defines browse-path entries, resolution results, and gates.
include/​ros2_medkit_opcua/​fault_code_lanes.hpp Introduces per-fault-code service-call sequencing.
include/​ros2_medkit_opcua/​address_space_browser.hpp Declares namespace lookup and path resolution interfaces.
CMakeLists.txt Registers new tests and certificate-generation support.
src/​ros2_medkit_fault_detection/​test/​test_fault_detection.cpp Tests multiple healthy enum states.
src/​ros2_medkit_fault_detection/​include/​ros2_medkit_fault_detection/​fault_detection.hpp Adds healthy-value lists to shared enum detection.

Review effort: Balanced

Comment on lines +1538 to +1542
if (node_id_index_.count(moved_to) > 0) {
RCLCPP_WARN(rclcpp::get_logger("opcua.node_map"),
"browse_path '%s' now resolves to %s, which another entry maps - keeping %s",
entry.browse_path_str.c_str(), moved_to.c_str(), entry.node_id_str.c_str());
continue;
Comment on lines +52 to +54
if (lane.busy && now - lane.since < stale_after_) {
lane.queue.push_back(std::move(send));
return;
Comment on lines +1492 to +1498
const auto index = std::stoul(prefix);
if (prefix.size() > 5 || index > 65535) {
error = "step " + std::to_string(i + 1) + ": namespace index " + prefix + " is out of range";
out.clear();
return false;
}
element.namespace_index = static_cast<uint16_t>(index);
j["write_capable"] = MEDKIT_OPCUA_READ_ONLY == 0;
j["poll_count"] = snap.poll_count;
j["error_count"] = snap.error_count;
j["last_error"] = client_->last_error();
Comment on lines +1994 to +1996
if (resolution.joined > 0) {
create_value_publishers();
}

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants