Skip to content

Recording manifest and per-file download routes, keep the recording just stored - #785

Merged
mfaferek93 merged 13 commits into
mainfrom
feat/recording-manifest
Nov 11, 2026
Merged

mfaferek93 merged 13 commits into
mainfrom
feat/recording-manifest

Conversation

@mfaferek93

@mfaferek93 mfaferek93 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

A recording split past snapshots.rosbag.max_bag_size_mb can now be downloaded in full, one storage file at a time, with resume.

  • Manifest. When a recording is finalized, the fault manager writes manifest.json next to metadata.yaml: the storage files in relative_file_paths order, the size and sha256 of each, and total_bytes. It is written to a temporary file, synced and renamed, before the recording row is stored.
  • Routes on the six entity paths:
    • GET .../bulk-data/rosbags/{recording_id}/manifest returns the manifest as stored.
    • GET .../bulk-data/rosbags/{recording_id}/files/{index} returns one storage file with Range support. A range that cannot be satisfied answers 416.
    • An unknown recording answers 404 with x-medkit-recording-not-found. A fault manager that does not answer gives 503, so a client can tell "gone" from "try again".
  • Listing. x-medkit.manifest: true when the manifest exists. The listing answers 503 when a list_rosbags call fails, instead of an empty 200.
  • Error bodies are not cut by the request's Range. A resumed request for a missing recording used to get an empty 416. This affects all binary download routes.
  • Quota sweep never removes the recording that was just written. When it alone exceeds max_total_storage_mb, a warning names it. A startup warning fires when max_bag_size_mb is larger than max_total_storage_mb.
  • Unique bag paths. Two captures with the same name no longer remove each other's directory.

The existing whole-recording download route is unchanged. max_bag_size_mb keeps its default of 50.

This touches the same files as #679, #684, #699 and #700 (rosbag_capture.cpp, bulkdata_handlers.cpp, rest.rst, snapshots.rst), so whichever lands second needs a rebase.


Changelog

Text for the CHANGELOGs when the release is cut.

ros2_medkit_fault_manager

  • A closed rosbag recording carries a manifest.json next to its storage files. It lists the files in the order metadata.yaml names them, each with its size and sha256, and their total size, so a recording split across several files past snapshots.rosbag.max_bag_size_mb can be fetched and checked as a whole. The manifest is written once, through a temporary file and a rename, after the writer is closed and before the recording is listed, and it is never rewritten. A recording whose manifest could not be written is still listed, without one, and recordings made before this change get none. The stored size_bytes of a recording includes the manifest.
  • A closed recording is hashed and stored by a worker the capture owns, not on the executor thread. The post-fault timer only closes the writer, so services keep answering and the capture keeps receiving while a large recording is hashed. stop() drains the worker. A fault cleared while its recording is being stored takes the row with it.
  • A recording larger than snapshots.rosbag.max_total_storage_mb is no longer deleted by its own quota sweep right after it is written. The sweep evicts oldest first and stopped only under the quota, so a recording that alone exceeded it, the newest row, went the same way as everything before it and the fault kept no black box at all. The recording just stored is now never evicted: every older recording may go, it stays, and a warning says the storage is over the limit. A warning is also logged at startup when max_bag_size_mb is larger than max_total_storage_mb.
  • A recording never opens inside an existing recording's directory. The directory name carries the millisecond the path was generated in and nothing made it unique, so a capture landing on a taken name failed to open, and its cleanup then removed the earlier recording. A taken name now moves the timestamp on by one millisecond until a free one is found, the name is reserved while the writer opens, and a failed open removes only the directory it created itself. The id format is unchanged.

ros2_medkit_gateway

  • Two routes fetch a rosbag recording that is split across several storage files. GET /{entity}/bulk-data/rosbags/{recording_id}/manifest returns the recording's manifest.json byte for byte: its files in order, each with size and sha256, and their total size. GET /{entity}/bulk-data/rosbags/{recording_id}/files/{index} serves one of those files by its position in the manifest, with Range support. Both take a recording id only, with no fallback to a fault code, and authorize like the existing download. A descriptor in the rosbags listing carries x-medkit.manifest: true when its recording has a manifest.
  • On those two routes a client can tell "the recording does not exist" from "not now". 404 with vendor code x-medkit-recording-not-found is sent only when the fault manager answered and holds no recording with that id. 503 is sent when the fault manager did not answer. Every other 404 (unknown entity, another entity's recording, no manifest, an index out of range, a file missing on disk) carries no such vendor code.
  • Binary downloads resolve a Range as RFC 9110 says, on the file route and on the existing download route alike. A last byte past the end is clamped to the file, a part that starts past the end is dropped, and only a request with no part left answers 416, with Content-Range: bytes */<size>. Content-Range always carries the absolute first and last byte, multipart parts included. A download that cannot seek ignores the Range and answers 200. cpp-httplib served a content provider without checking the range, so a first byte past the end produced a 206 with an underflowed Content-Length.
  • An error answer is no longer cut by the request's Range. cpp-httplib applies the range to whatever body a response carries, so a client resuming a download got the error document as a slice of itself, or an empty 416 when its offset was past the end of that document, in place of the real status. This covers every 4xx and 5xx, including the 401, 403 and 429 answered before routing. The same applied to a ranged response an aggregating gateway relays from a peer, which was ranged a second time.
  • Breaking: GET /{entity}/bulk-data/rosbags answers 503 when the fault manager cannot be read for any of the sources the entity covers. A failed read used to be skipped, so the listing answered 200 with those recordings missing, or with an empty collection when every read failed, and a client could not tell "no recordings" from "no answer". A 200 listing is now complete.

Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

Built and tested in Docker on Jazzy (Release, 0 compiler warnings in the touched packages):

  • colcon test for ros2_medkit_fault_manager (36/36), ros2_medkit_gateway (143/143), ros2_medkit_integration_tests (140/140).
  • New: test_recording_manifest.cpp (13), test_rosbag_recording.cpp (36, lookup, Range, index, handlers, listing 503), 8 new cases in test_rosbag_capture.cpp, integration tests test_rosbag_recording_manifest.test.py (a real 16 MB split into 1 MB files) and test_rosbag_recording_unavailable.test.py (no fault manager).
  • clang-format-18 --dry-run --Werror on the changed files, Sphinx -W build clean.

After review: the same three packages pass again (fault_manager 36/36, gateway 143/143). test_rosbag_recording_manifest.test.py now runs over a 1 MB quota and waits on events, test_range_on_refused_requests.test.py covers the 401, 403 and 429 under a Range, and AClearWhileTheRowIsStoredTakesTheRowWithIt and StoringARecordingDoesNotHoldTheExecutor cover the store worker. The tutorial's reindex step was run on a real 11-file download on Jazzy.

Manual run with a real capture (ros2 bag play --loop of an 11-topic bag, max_bag_size_mb: 50): recordings of 232 MB (5 files) to 498 MB (10 files). For each, the manifest from the route equals the file on disk, every file's download sha256 equals the manifest, every file fetched in two Range parts and joined matches, out-of-range requests give 416, and with the fault manager stopped the listing, manifest and file routes give 503.


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

A recording larger than max_total_storage_mb was evicted by its own
quota sweep right after it was written. A bag name that was already
taken made the open fail, and the cleanup removed the earlier recording.
The fault manager writes manifest.json when it closes a recording. The
gateway serves it and each storage file by index with Range, and tells
a missing recording (404 with a vendor code) from no answer (503).
The rosbags listing answers 503 when it cannot be read in full.
… Range

cpp-httplib applied the Range to the JSON body and answered 200 with a slice.
JSON document routes now ignore the header. Also clears three clang-tidy
warnings in the bag path code and tags the new recording tests.
…llowed

The manifest is written by hand in its documented key order, so the ROS-free
core builds with nlohmann-json 3.7. A symbolic link in a recording directory is
neither listed nor served. A declined lookup for an entity with no source is a
plain 404: nothing proves the recording absent.
The example listed one file under the total of twelve, against the rule
stated right below it. It now lists two files that add up.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 20:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Comment thread docs/api/rest.rst Resolved
Comment thread docs/api/rest.rst Resolved
Comment thread docs/tutorials/snapshots.rst Resolved
The post-fault timer closed the recording, hashed every storage file and
stored its rows on the only executor thread, so no service answered and
the capture dropped messages meanwhile. A worker owned by the capture now
writes the manifest, then the rows, and is drained by stop().
The recording is larger than max_total_storage_mb, so the real fault
manager's sweep runs on it: the test checks it is kept and that the
"alone exceeds" warning is logged. The fixed 4 s wait becomes a wait for
the capture's subscription and for the publisher to have sent enough.
cpp-httplib applies the request's ranges to every error body right after
the global error handler, so a resume with an expired token got an empty
416 instead of its 401. The handler now clears them for every 4xx and 5xx,
pre-routing 401, 403 and 429 included.
binary_download resolves the Range against the file: a last byte past the
end is clamped, a part starting past it is dropped, Content-Range is an
absolute first-last, and only a request with nothing left gets a 416, with
Content-Range: bytes */size. A provider that cannot seek ignores the Range.
The downloaded files have no metadata.yaml, so ROS 2 cannot open the
directory. ros2 bag reindex writes it again, and the directory plays.
Changelogs are written on the release branch when the release is cut.
The text moves to the PR description.
The store worker checked that a fault was not cleared, then inserted its
row. A clear landing between the two deleted nothing and the row stayed.
The check and the insert now hold one mutex with on_fault_cleared()'s
delete. Hashing stays outside it.
The racer kept confirming B after its boundary bag opened, so a second
recording of B started once that one closed. With the store off the
executor it can hold buffered history, and the duration check read it.
The racer now stops at the boundary bag, and B must have one recording.
@mfaferek93
mfaferek93 merged commit b8f1fc6 into main Nov 11, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants