Repository navigation
fix: report a rosbag's served size, retire the removed snapshot routes - #679
Merged
Merged
Conversation
mfaferek93
reviewed
Sep 14, 2026
bburda
force-pushed
the
fix/bulk-data-size-and-stale-snapshot-routes
branch
3 times, most recently
from
September 20, 2026 18:47
df614a6 to
0fc66e3
Compare
mfaferek93
reviewed
Sep 28, 2026
mfaferek93
reviewed
Nov 9, 2026
4 of 7 tasks
bburda
force-pushed
the
fix/bulk-data-size-and-stale-snapshot-routes
branch
from
November 10, 2026 20:36
0fc66e3 to
f8a8280
Compare
A rosbag2 recording is a directory with one storage file and
metadata.yaml. The download route resolves the storage file and streams
only that file. The listing reported the figure the fault manager
stores, which is the size of the whole directory. The two numbers answer
different questions. The stored figure is the footprint of the
recording against its disk quota. The listed figure is what a client is
about to fetch. So every listing overstated the download, on a short
recording by about a tenth of the transfer. A client that sized a
buffer or a progress bar from the listing never reached the end.
The listing now measures the file that the download resolves, through
the same resolver. The promised length is the length that arrives.
resolve_rosbag_file_path becomes a public static member so the listing
can call it. When this process cannot see the bag, the descriptor keeps
the stored figure. It is the only number available, and a zero would
describe the recording as empty. The stored figure itself is unchanged,
so quota accounting still counts the bytes the recording occupies.
rest.rst and the DTO comment state that size is the byte count the
download serves.
Also drop the removed snapshot endpoints from the gateway README and
from the quick start of the snapshots tutorial. GET
/faults/{code}/snapshots and .../snapshots/bag answer 404. A fault
returns its snapshots inline, and recordings are downloaded through the
bulk-data endpoints. The migration table in the tutorial stays, because
it points a reader at the replacements.
A recording is stored as a directory and served as a single file, so it has two sizes. The row carried one number for both jobs: the directory total. That total is what the recording costs against max_total_storage_mb. Every API answer that quoted it overstated the download by metadata.yaml, on a short recording by about a tenth of the transfer. The environment_data.snapshots[] entry next to a download link was the worst case, because a client sizes its transfer from that number. rosbag_served_bytes() now answers the reporting question on its own. It measures the storage file that the bag's metadata.yaml names. It reads the metadata through rosbag2_storage::MetadataIo, the library that wrote it. GetFault, GetSnapshots, GetRosbag and ListRosbags report their size through it. RosbagFileInfo::size_bytes and the quota are unchanged. They still count the whole directory, which is what eviction frees. When no single served file can be named, the function returns the stored total. This covers a missing metadata.yaml, one that cannot be parsed, a named file that is gone, and a recording split across several storage files past the maximum bag size. None of these is an error. The stored total is a real measurement of the recording. A zero would describe it as empty. rest.rst now states the rule once. The descriptor size, the nested size_bytes and the Content-Length of the download are the same number, and that number is the size of the storage file. A split recording is the exception and reports its total.
… payload
The GetSnapshots response carried rosbag.download_url, built as
/api/v1/faults/{code}/snapshots/bag. That route was removed in 0.2.0, so
the field gave every caller a URL that answers 404.
No code in this repository reads the field. The gateway builds its own
entity-scoped bulk-data URI from the recording id it receives on the
GetFault snapshot entries.
The field is removed, and this payload gets no new URL. A recording is
addressed under its entity, as
/api/v1/{entity-type}/{id}/bulk-data/rosbags/{recording_id}. Which of
the four entity types owns a given source is part of the gateway's
discovery model. The fault manager holds the recording id and the
reporting source, but it does not hold that mapping. A URL built here
would be a guess at one of four prefixes, and a wrong URL that looks
right misleads the caller.
Service-level tests now cover the size that GetSnapshots, GetRosbag and
ListRosbags report, next to the existing GetFault test. A shared bag
fixture backs them. The test helpers call file_size() without a cast.
It already returns uintmax_t, so the cast was an identity cast that
-Wuseless-cast flags.
…unreadable bag Sizing rule. For a bag directory, the listing measures a file only when metadata.yaml names exactly one storage file, the name is a direct child of the bag with a .db3 or .mcap extension, and the file is present. In every other case the descriptor keeps the row's figure, which is the fault manager's answer for the same recording. That covers a split recording, a missing or unparsable metadata.yaml, a name that escapes the directory, and a named file that is gone. The fault manager decides "split" from the same relative_file_paths field, so both API surfaces report one size for one recording. A path that is itself a storage file is one storage file, and the listing sizes it as it is. A split recording keeps its total because the download hands over one segment, and no single file describes the transfer. The descriptor size is then larger than the Content-Length of the download. rest.rst states this as the way a client recognises a partial download of a split recording. The Content-Length entry links to that rule with explicit link text, because the label sits on a paragraph. Resolver. When the metadata names one storage file under the same rule and the file exists, the resolver returns that file. Otherwise it scans the directory for the first .db3 or .mcap, so the download always has a file to hand over when the directory holds one. A stray file next to the recording, such as a leftover segment or a copy, is never sized as the recording. The gateway reads relative_file_paths with yaml-cpp, which it already links. It does not link rosbag2_storage. Filesystem errors. Every filesystem call in the resolver and in the size helper uses the std::error_code overload. EACCES on one bag directory, or ENOENT when quota eviction removes a bag during the walk, now leaves that one recording unmeasured. The listing handler has no catch in its chain, so the exception answered 500 and dropped every other recording of the entity. The download route shares the resolver and gets the same protection. The comments on the resolver and on rosbag_served_bytes state when the listing and the download report the same length. README. The download is one storage file streamed verbatim, named <recording_id>.<format> (.mcap or .sqlite3). It is not an archive and it does not include metadata.yaml. The playback section runs ros2 bag info and ros2 bag play on the downloaded file, with no unpacking step and no --storage flag, for both storage formats. The Postman collection drops its three requests to the snapshot routes that were removed in 0.2.0. The collection has no bulk-data section to move them to. Tests. A new integration test compares a descriptor's size with the bytes its own download delivers and with Content-Length. Unit tests cover a split recording, an unreadable directory, a bare storage file, a stray file, a named file that is gone, a name that climbs out of the bag, an absolute name, a name in a subdirectory and a name without a storage extension. The stray-file test names the storage file that the directory scan would not reach first. The mode-0000 test drops CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH from its own thread's effective set when it runs as root. It restores them through a guard object, and it asserts that the permission is denied before it asserts anything about the resolver. A test uses the result of a nodiscard is_directory call. A fault manager test pins the same 4096-byte answer for a named recording with a stray file beside it.
…side it rosbag_served_bytes sizes the named file only when the name is a direct child of the bag directory with a .db3 or .mcap extension; any other name falls back to the stored total, the same rule the gateway applies. The stray-file test's cross-reference names that rule.
rest.rst now gives the full condition. The descriptor size, size_bytes and Content-Length are one number when metadata.yaml names exactly one storage file, the name is a direct child of the bag with a .db3 or .mcap extension, and that file is present. The page lists the four cases where the reported size is the stored total while the download hands over one file. It states that a bag directory with no storage file answers 500. It also corrects the on-disk extension of a sqlite3 recording from .sqlite3 to .db3.
…t in the core The served-size helper moves into the ROS-free core as core::rosbag_served_bytes. It reads only rosbag2_bagfile_information.relative_file_paths with yaml-cpp, under the same rules as the gateway. rosbag2_storage::MetadataIo decoded the whole document and rejected documents the gateway accepted, so the two sides could report different sizes for one bag. A metadata.yaml or named storage file that is a symlink is not followed. GetFault reports the served size again. get_locked collects the bag paths and get() measures them after it releases the core mutex, so a slow disk does not hold up fault reports.
The resolver, the directory scan and the listing size accept a bag's metadata.yaml and storage file only when they are regular files. They read metadata.yaml from the normalised bag path, as the fault manager does. detail::rosbag_served_bytes stats the named file directly and no longer calls the resolver. The download opens the served file once with O_NOFOLLOW, checks on that descriptor that it is a regular file, and serves every range from it with pread. A file swapped for a symlink after resolving is refused. The integration test swaps a live recording's storage file for a symlink to a file outside the bag and checks the listing, the fault snapshot entry and the download.
The size_bytes comments in Snapshot.msg, GetRosbag.srv and ListRosbags.srv say the field is the served storage file, and list when the directory total is reported. rest.rst scopes the size promise to recordings that meet the rule, says when the stored total is reported, and names the symlink cases. The gateway README says Humble needs the mcap storage plugin installed.
Return the named path by value so it can be moved, delete the move members of the two test bag fixtures, and use the char overload of find for one-letter searches.
…files A recording split past snapshots.rosbag.max_bag_size_mb was reported at its stored directory total, which counts metadata.yaml and manifest.json. No route serves those two files, so a client that fetched every storage file never reached the reported size. The reported size is now the sum of the storage files metadata.yaml names. For a single-file recording that is the file the download serves, as before. For a split it equals the manifest's total_bytes. The stored total stays the figure the quota counts, and it is still reported when the metadata cannot be read or a named file is not a regular .db3 or .mcap inside the bag.
bburda
force-pushed
the
fix/bulk-data-size-and-stale-snapshot-routes
branch
from
November 11, 2026 16:01
f8a8280 to
aaf75b2
Compare
mfaferek93
approved these changes
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A rosbag2 recording is a directory with one storage file plus
metadata.yaml. The bulk-data download serves the storage file alone, but every size in the API was the size of the whole directory, so each listing overstated its own download.Now:
sizeunder/bulk-data/rosbagsis the size of the file the download serves, when the bag'smetadata.yamlnames one storage file.GetFault,GetSnapshots,GetRosbagandListRosbagsis the sum of the storage files thatmetadata.yamlnames. For a recording split pastmax_bag_size_mb, this equals the manifest'stotal_bytesfrom Recording manifest and per-file download routes, keep the recording just stored #785. Themax_total_storage_mbquota still counts the whole directory, because that is what eviction frees..db3or.mcapfile directly inside the bag, and the download opens it withO_NOFOLLOW.docs/api/rest.rstsays when the descriptorsize, the fault'ssize_bytesandContent-Lengthare the same number, and lists the cases where they differ.Removed: the README and tutorial sections for
GET /faults/{code}/snapshotsand.../snapshots/bag(both answer 404), the README step that rantar -xzfon the download, three postman requests to the removed routes, androsbag.download_urlin theGetSnapshotsresponse.Issue
Type
Breaking: the
GetSnapshotsservice response no longer carriesrosbag.download_url. The REST API shape is unchanged.Testing
test_bulk_data_apichecks that the descriptorsizeequals the length of the download.test_rosbag_recording_manifestrecords a real split recording and checks that the listingsizeand the fault'ssize_bytesequal the manifest'stotal_bytes.Checklist