Skip to content

fix: report a rosbag's served size, retire the removed snapshot routes - #679

Merged
bburda merged 11 commits into
mainfrom
fix/bulk-data-size-and-stale-snapshot-routes
Nov 12, 2026
Merged

bburda merged 11 commits into
mainfrom
fix/bulk-data-size-and-stale-snapshot-routes

Conversation

@bburda

@bburda bburda commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A rosbag2 recording is a directory with one storage file plus metadata.yaml. The bulk-data download serves the storage file alone, but every size in the API was the size of the whole directory, so each listing overstated its own download.

Now:

  • The descriptor size under /bulk-data/rosbags is the size of the file the download serves, when the bag's metadata.yaml names one storage file.
  • The size the fault manager reports through GetFault, GetSnapshots, GetRosbag and ListRosbags is the sum of the storage files that metadata.yaml names. For a recording split past max_bag_size_mb, this equals the manifest's total_bytes from Recording manifest and per-file download routes, keep the recording just stored #785. The max_total_storage_mb quota still counts the whole directory, because that is what eviction frees.
  • When the named files cannot be measured (missing or broken metadata, a named file that is gone, a name outside the bag or a symbolic link), the size falls back to the stored total. The gateway and the fault manager follow a name only to a regular .db3 or .mcap file directly inside the bag, and the download opens it with O_NOFOLLOW.
  • A bag that cannot be read keeps its stored size, and the rest of the listing still answers.
  • docs/api/rest.rst says when the descriptor size, the fault's size_bytes and Content-Length are the same number, and lists the cases where they differ.

Removed: the README and tutorial sections for GET /faults/{code}/snapshots and .../snapshots/bag (both answer 404), the README step that ran tar -xzf on the download, three postman requests to the removed routes, and rosbag.download_url in the GetSnapshots response.

Issue

Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Breaking: the GetSnapshots service response no longer carries rosbag.download_url. The REST API shape is unchanged.

Testing

colcon build --packages-up-to ros2_medkit_gateway ros2_medkit_fault_manager ros2_medkit_integration_tests
colcon test --packages-select ros2_medkit_gateway --ctest-args -R test_bulkdata_handlers
colcon test --packages-select ros2_medkit_fault_manager --ctest-args -R 'test_rosbag_capture|test_fault_manager'
colcon test --packages-select ros2_medkit_integration_tests --ctest-args -R 'test_bulk_data_api|test_rosbag_recording_manifest'
colcon test-result --verbose

test_bulk_data_api checks that the descriptor size equals the length of the download. test_rosbag_recording_manifest records a real split recording and checks that the listing size and the fault's size_bytes equal the manifest's total_bytes.

Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

@bburda bburda self-assigned this Sep 10, 2026
@bburda
bburda marked this pull request as ready for review September 12, 2026 17:31
@bburda
bburda force-pushed the fix/bulk-data-size-and-stale-snapshot-routes branch 3 times, most recently from df614a6 to 0fc66e3 Compare September 20, 2026 18:47
Comment thread docs/api/rest.rst Resolved
Comment thread src/ros2_medkit_gateway/README.md Resolved
@bburda
bburda force-pushed the fix/bulk-data-size-and-stale-snapshot-routes branch from 0fc66e3 to f8a8280 Compare November 10, 2026 20:36
@bburda
bburda added this pull request to stack #786 November 11, 2026 12:02
A rosbag2 recording is a directory with one storage file and
metadata.yaml. The download route resolves the storage file and streams
only that file. The listing reported the figure the fault manager
stores, which is the size of the whole directory. The two numbers answer
different questions. The stored figure is the footprint of the
recording against its disk quota. The listed figure is what a client is
about to fetch. So every listing overstated the download, on a short
recording by about a tenth of the transfer. A client that sized a
buffer or a progress bar from the listing never reached the end.

The listing now measures the file that the download resolves, through
the same resolver. The promised length is the length that arrives.
resolve_rosbag_file_path becomes a public static member so the listing
can call it. When this process cannot see the bag, the descriptor keeps
the stored figure. It is the only number available, and a zero would
describe the recording as empty. The stored figure itself is unchanged,
so quota accounting still counts the bytes the recording occupies.
rest.rst and the DTO comment state that size is the byte count the
download serves.

Also drop the removed snapshot endpoints from the gateway README and
from the quick start of the snapshots tutorial. GET
/faults/{code}/snapshots and .../snapshots/bag answer 404. A fault
returns its snapshots inline, and recordings are downloaded through the
bulk-data endpoints. The migration table in the tutorial stays, because
it points a reader at the replacements.
A recording is stored as a directory and served as a single file, so it
has two sizes. The row carried one number for both jobs: the directory
total. That total is what the recording costs against
max_total_storage_mb. Every API answer that quoted it overstated the
download by metadata.yaml, on a short recording by about a tenth of the
transfer. The environment_data.snapshots[] entry next to a download link
was the worst case, because a client sizes its transfer from that
number.

rosbag_served_bytes() now answers the reporting question on its own. It
measures the storage file that the bag's metadata.yaml names. It reads
the metadata through rosbag2_storage::MetadataIo, the library that wrote
it. GetFault, GetSnapshots, GetRosbag and ListRosbags report their size
through it. RosbagFileInfo::size_bytes and the quota are unchanged. They
still count the whole directory, which is what eviction frees.

When no single served file can be named, the function returns the
stored total. This covers a missing metadata.yaml, one that cannot be
parsed, a named file that is gone, and a recording split across several
storage files past the maximum bag size. None of these is an error. The
stored total is a real measurement of the recording. A zero would
describe it as empty.

rest.rst now states the rule once. The descriptor size, the nested
size_bytes and the Content-Length of the download are the same number,
and that number is the size of the storage file. A split recording is
the exception and reports its total.
… payload

The GetSnapshots response carried rosbag.download_url, built as
/api/v1/faults/{code}/snapshots/bag. That route was removed in 0.2.0, so
the field gave every caller a URL that answers 404.

No code in this repository reads the field. The gateway builds its own
entity-scoped bulk-data URI from the recording id it receives on the
GetFault snapshot entries.

The field is removed, and this payload gets no new URL. A recording is
addressed under its entity, as
/api/v1/{entity-type}/{id}/bulk-data/rosbags/{recording_id}. Which of
the four entity types owns a given source is part of the gateway's
discovery model. The fault manager holds the recording id and the
reporting source, but it does not hold that mapping. A URL built here
would be a guess at one of four prefixes, and a wrong URL that looks
right misleads the caller.

Service-level tests now cover the size that GetSnapshots, GetRosbag and
ListRosbags report, next to the existing GetFault test. A shared bag
fixture backs them. The test helpers call file_size() without a cast.
It already returns uintmax_t, so the cast was an identity cast that
-Wuseless-cast flags.
…unreadable bag

Sizing rule. For a bag directory, the listing measures a file only when
metadata.yaml names exactly one storage file, the name is a direct child
of the bag with a .db3 or .mcap extension, and the file is present. In
every other case the descriptor keeps the row's figure, which is the
fault manager's answer for the same recording. That covers a split
recording, a missing or unparsable metadata.yaml, a name that escapes
the directory, and a named file that is gone. The fault manager decides
"split" from the same relative_file_paths field, so both API surfaces
report one size for one recording. A path that is itself a storage
file is one storage file, and the listing sizes it as it is.

A split recording keeps its total because the download hands over one
segment, and no single file describes the transfer. The descriptor size
is then larger than the Content-Length of the download. rest.rst states
this as the way a client recognises a partial download of a split
recording. The Content-Length entry links to that rule with explicit
link text, because the label sits on a paragraph.

Resolver. When the metadata names one storage file under the same rule
and the file exists, the resolver returns that file. Otherwise it scans
the directory for the first .db3 or .mcap, so the download always has a
file to hand over when the directory holds one. A stray file next to the
recording, such as a leftover segment or a copy, is never sized as the
recording. The gateway reads relative_file_paths with yaml-cpp, which it
already links. It does not link rosbag2_storage.

Filesystem errors. Every filesystem call in the resolver and in the size
helper uses the std::error_code overload. EACCES on one bag directory,
or ENOENT when quota eviction removes a bag during the walk, now leaves
that one recording unmeasured. The listing handler has no catch in its
chain, so the exception answered 500 and dropped every other recording
of the entity. The download route shares the resolver and gets the same
protection.

The comments on the resolver and on rosbag_served_bytes state when the
listing and the download report the same length.

README. The download is one storage file streamed verbatim, named
<recording_id>.<format> (.mcap or .sqlite3). It is not an archive and it
does not include metadata.yaml. The playback section runs ros2 bag info
and ros2 bag play on the downloaded file, with no unpacking step and no
--storage flag, for both storage formats.

The Postman collection drops its three requests to the snapshot routes
that were removed in 0.2.0. The collection has no bulk-data section to
move them to.

Tests. A new integration test compares a descriptor's size with the
bytes its own download delivers and with Content-Length. Unit tests
cover a split recording, an unreadable directory, a bare storage file, a
stray file, a named file that is gone, a name that climbs out of the
bag, an absolute name, a name in a subdirectory and a name without a
storage extension. The stray-file test names the storage file that the
directory scan would not reach first. The mode-0000 test drops
CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH from its own thread's
effective set when it runs as root. It restores them through a guard
object, and it asserts that the permission is denied before it asserts
anything about the resolver. A test uses the result of a nodiscard
is_directory call. A fault manager test pins the same 4096-byte answer
for a named recording with a stray file beside it.
…side it

rosbag_served_bytes sizes the named file only when the name is a direct
child of the bag directory with a .db3 or .mcap extension; any other name
falls back to the stored total, the same rule the gateway applies. The
stray-file test's cross-reference names that rule.
rest.rst now gives the full condition. The descriptor size, size_bytes
and Content-Length are one number when metadata.yaml names exactly one
storage file, the name is a direct child of the bag with a .db3 or .mcap
extension, and that file is present. The page lists the four cases
where the reported size is the stored total while the download hands
over one file. It states that a bag directory with no storage file
answers 500. It also corrects the on-disk extension of a sqlite3
recording from .sqlite3 to .db3.
…t in the core

The served-size helper moves into the ROS-free core as
core::rosbag_served_bytes. It reads only
rosbag2_bagfile_information.relative_file_paths with yaml-cpp, under the same
rules as the gateway. rosbag2_storage::MetadataIo decoded the whole document and
rejected documents the gateway accepted, so the two sides could report
different sizes for one bag. A metadata.yaml or named storage file that is a
symlink is not followed.

GetFault reports the served size again. get_locked collects the bag paths and
get() measures them after it releases the core mutex, so a slow disk does not
hold up fault reports.
The resolver, the directory scan and the listing size accept a bag's
metadata.yaml and storage file only when they are regular files. They read
metadata.yaml from the normalised bag path, as the fault manager does.
detail::rosbag_served_bytes stats the named file directly and no longer calls
the resolver.

The download opens the served file once with O_NOFOLLOW, checks on that
descriptor that it is a regular file, and serves every range from it with
pread. A file swapped for a symlink after resolving is refused.

The integration test swaps a live recording's storage file for a symlink to a
file outside the bag and checks the listing, the fault snapshot entry and the
download.
The size_bytes comments in Snapshot.msg, GetRosbag.srv and ListRosbags.srv
say the field is the served storage file, and list when the directory total is
reported. rest.rst scopes the size promise to recordings that meet the rule,
says when the stored total is reported, and names the symlink cases. The
gateway README says Humble needs the mcap storage plugin installed.
Return the named path by value so it can be moved, delete the move members of
the two test bag fixtures, and use the char overload of find for one-letter
searches.
…files

A recording split past snapshots.rosbag.max_bag_size_mb was reported at its
stored directory total, which counts metadata.yaml and manifest.json. No route
serves those two files, so a client that fetched every storage file never
reached the reported size.

The reported size is now the sum of the storage files metadata.yaml names. For
a single-file recording that is the file the download serves, as before. For a
split it equals the manifest's total_bytes. The stored total stays the figure
the quota counts, and it is still reported when the metadata cannot be read or
a named file is not a regular .db3 or .mcap inside the bag.
@bburda
bburda force-pushed the fix/bulk-data-size-and-stale-snapshot-routes branch from f8a8280 to aaf75b2 Compare November 11, 2026 16:01
@bburda
bburda merged commit 91a587c into main Nov 12, 2026
40 of 68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bulk-data rosbags: the descriptor size counts bytes the download does not serve, and the README documents removed snapshot routes

2 participants