Skip to content

fix(gateway): download a split recording's first file, and say how many there are - #684

Merged
bburda merged 6 commits into
fix/bulk-data-size-and-stale-snapshot-routesfrom
fix/split-recording-first-segment-and-file-count
Nov 12, 2026
Merged

bburda merged 6 commits into
fix/bulk-data-size-and-stale-snapshot-routesfrom
fix/split-recording-first-segment-and-file-count

Conversation

@bburda

@bburda bburda commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

A recording split past snapshots.rosbag.max_bag_size_mb downloaded as whichever storage file the directory listing gave first, and the descriptor did not say that the recording had more files.

When a bag's metadata.yaml names storage files, GET .../bulk-data/rosbags/{recording_id} now serves the first named file, where the recording starts. If that file is missing, is a symbolic link, or the name is not a .db3 or .mcap directly inside the bag, the download answers 500. It never serves a later segment or another file in its place. The directory scan stays only for a bag with no readable metadata or an empty file list.

The descriptor's x-medkit block carries storage_files, the number of files the metadata names. It is omitted when the metadata does not say. When it is more than 1 and the descriptor carries "manifest": true, a client fetches the whole recording through the manifest and file routes from #785. The descriptor size is the sum of the storage files (#679), so for a split recording it is larger than the download's Content-Length. docs/api/rest.rst and the snapshots tutorial describe the rule.

The listing reads metadata.yaml once per row and takes the served file, the count and the size from that read.

Three tests could fail at random, also on main. They are fixed here:

  • test_graph_node_list now waits until the graph lists the probe node's endpoints. The rcl graph learns them asynchronously, so the check could run too early.
  • test_mutual_peer_faults now tells the three peer gateways' faults apart by reporting_sources. It used the fault time, and two fault managers can stamp the same time. Each peer's lidar runs in its own namespace for this.
  • test_domain_wrapper could read a command's domain file while it was still empty. The commands now write the number to a temporary file and rename it into place.

Stacked on #679.


Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

colcon build --packages-up-to ros2_medkit_gateway ros2_medkit_integration_tests
colcon test --packages-select ros2_medkit_gateway --ctest-args -R 'test_bulkdata_handlers|test_graph_node_list'
colcon test --packages-select ros2_medkit_integration_tests --ctest-args -R 'test_rosbag_recording_manifest|test_bulk_data_api|test_mutual_peer_faults'
colcon test-result --verbose

test_rosbag_recording_manifest records a real split recording. It checks that the download hands over the manifest's first file, by sha256, and that storage_files equals the number of files in the manifest. With the first file moved aside and a later .mcap still in the bag, the download answers 500.


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

@bburda bburda self-assigned this Sep 10, 2026
Comment thread docs/api/rest.rst Resolved
Comment thread src/ros2_medkit_gateway/CHANGELOG.rst Resolved
@bburda
bburda force-pushed the fix/split-recording-first-segment-and-file-count branch from 3fea8bb to 28c623d Compare November 11, 2026 12:02
@bburda
bburda added this pull request to stack #786 November 11, 2026 12:02
@bburda
bburda marked this pull request as ready for review November 11, 2026 13:21
…ny there are

A recording split past snapshots.rosbag.max_bag_size_mb downloaded as whichever
storage file the directory listing yielded first, and nothing in the descriptor
said the recording had more.

When a bag's metadata.yaml names storage files, the download now serves the
first name and nothing else. If that file is missing, or the name is not a .db3
or .mcap directly inside the bag, the download answers 500. A later segment or
a stray file is never served in its place. The directory scan is left only for
a bag with no readable metadata or an empty file list.

The rosbag descriptor's x-medkit block carries storage_files, the number of
files the metadata names. It is omitted when the metadata does not say. The
listing reads metadata.yaml once per row and takes the served file, the count
and the size from that read.
@bburda
bburda force-pushed the fix/split-recording-first-segment-and-file-count branch from 28c623d to 06fb4b3 Compare November 11, 2026 16:45
…ing it

GraphNodeHasEndpoints asserted that the graph lists the node's endpoints right
after the node was created. The rcl graph learns them asynchronously, through
discovery of the node's own participant, so the check could run before they
were listed and fail. The test now waits on the graph event, with a deadline,
until the node's endpoints are listed.
…r source

test_mutual_peer_faults identified each gateway's LIDAR_RANGE_INVALID by its
first_occurred time and required the three to differ. Each fault manager stamps
the fault with its own clock when the report arrives, and the three faulty
lidars start and report together, so two gateways can hold the same time.

Each gateway's lidar now runs in a namespace of its own, and the faults are
identified by reporting_sources. create_demo_nodes takes an optional namespace
that replaces the registry's for the nodes it launches.
On a status mismatch get_raw built its message from response.text. For a
binary download of a few megabytes, requests then guesses the charset, which
takes minutes, and the test hits its timeout with no message. The message now
carries the first 1024 bytes of the body, decoded as UTF-8.
mfaferek93
mfaferek93 previously approved these changes Nov 11, 2026
test_rosbag_recording_manifest now checks on its real split recording that
the recording download serves the first file the manifest lists, and that
x-medkit.storage_files counts every file. With the first file moved aside and
a later .mcap still in the bag, the download answers 500.

The docs say that the download serves the first file of a split recording,
and that a recording with a manifest is fetched file by file through the
manifest and file routes.
@bburda
bburda force-pushed the fix/split-recording-first-segment-and-file-count branch from e5fd9cd to eb881eb Compare November 12, 2026 07:09
The domain wrapper tests start commands that write their ROS_DOMAIN_ID to a
marker file, and the test reads the number as soon as the file exists.
write_text and the shell's > create the file empty before they write, so the
test could read an empty string and fail with ValueError. The commands now
write to <marker>.part and rename it into place, so the marker never exists
empty.
@bburda
bburda merged commit 82765f6 into main Nov 12, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants