I have dug through noisy logs many times just to find the one real cause. Two situations keep coming back.
At startup, every robot I have worked with throws a pile of errors because drivers are still connecting and nodes are waiting for each other. Most go away on their own, but they sit in the log next to the one that matters.
Then there are chain reactions. One thing breaks and a minute later other nodes report missing data, timeouts and a failed bringup. I already know this is really one fault, but the robot reports it as many.

The case we rebuilt
We staged a startup with Nav2 Jazzy and AMCL in which a Hokuyo Ethernet lidar never connects, as with a pulled data cable or a wrong subnet. The texts are verbatim from urg_node and Nav2, replayed at their real repeat rates.
urg_node reports the diagnostic status “Not Connected”. AMCL publishes map to odom only after it processes a scan, so it never publishes it here. It logs nothing about the missing scans.
global_costmap logs “Timed out waiting for transform from base_link to map to become available” every half second at INFO and fails to activate after 60 seconds. lifecycle_manager_navigation reports “Failed to bring up all requested nodes. Aborting bringup.” and bt_navigator rejects the goal.
The launch log ends up with 164 messages on 222 lines. The lidar error is in there 29 times, between 120 costmap messages. Nothing in the log links one to the other.
How AUTOSAR handles this
Cars have had this for years. The AUTOSAR Classic Diagnostic Event Manager has event dependencies (SWS DEM R24-11, section 7.3.6). Events are assigned to components with parent relations and priorities. A FAILED report while a parent component is FAILED is a “consecutive fault” and is not stored in the failure memory. A causality delay time lets a failure be reclassified when a parent or higher priority failure is reported shortly after it.
The same idea in ros2_medkit
For startup faults, when healing is enabled, PASSED reports heal a fault and take it off the active list.
For the chain reaction we wrote one rule. It names the lidar fault as the cause, the scan, costmap, lifecycle and goal rejection codes as its symptoms, and a time window:
correlation:
enabled: true
default_window_ms: 500
patterns:
scan_dependent_localization:
codes: ["LIDAR_SCAN_TOPIC_NO_EVENTS"]
nav2_global_bringup:
codes: ["GLOBAL_COSTMAP_*", "NAV_LIFECYCLE_*", "BT_NAVIGATOR_GOAL_REJECTED"]
rules:
- id: lidar_not_connected_blocks_localization
name: "Lidar driver is not connected"
mode: hierarchical
window_ms: 5000
root_cause:
codes: ["LIDAR_NOT_CONNECTED"]
symptoms:
- pattern: scan_dependent_localization
- pattern: nav2_global_bringup
mute_symptoms: true
auto_clear_with_root: true
A symptom that arrives after a report of the cause and inside the window is linked to it and muted. The window is measured from the latest report of the cause. That matters here because the costmap gives up a full minute after the lidar first fails. The rule works on fault codes that nodes report to the fault manager, so the reporters decide what a code means.
With this rule, GET /api/v1/faults returned one fault, LIDAR_NOT_CONNECTED, with muted_count 6. ?include_muted=true lists all seven with a root_cause_code for each symptom. Unlike AUTOSAR’s failure memory, a muted symptom stays stored.
Suggesting rules from fault history
After we first shared this, someone asked whether the rules have to stay hand-written (related comment). The fault history already holds what is needed to suggest them: which codes always follow which and how long after.
We would like the system to propose a rule once a pattern holds across many bringups, with an engineer approving it. A wrong rule would mute a real fault. The same history could also flag a symptom that shows up without its usual cause, which often means something changed on the robot. None of this exists yet.
Repo: GitHub - selfpatch/ros2_medkit: ros2_medkit - diagnostics gateway for ROS 2 robots. Faults, live data, operations, scripts, locking, triggers, and OTA updates via REST API. No SSH, no custom tooling. · GitHub
API responses and how to run the example: One Cause, Many Faults: Fault Correlation in ROS 2 | selfpatch.ai
Two questions for the packages you maintain:
- Which failures report their symptoms before the underlying cause, and how do you handle that ordering today?
- And would you trust a rule that was suggested from history, if you had to approve it first?
1 post - 1 participant
Read full topic