Repository navigation
initramfs: Enforce fs-verity for strict composefs boots - #2629
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
Changes to the early-boot integrity trust boundary and an unreleased composefs dependency require human security review and privileged boot validation across supported kernels and legacy repositories that could not be performed here.
No open issues 🎉
What changed in this PR
This PR hardens composefs boot integrity by making the kernel command line—not mutable repository metadata—control fs-verity enforcement in the initramfs.
Changes:
- Mount images by verified digest with an explicit integrity policy, excluding named-reference fallback.
- Add unit and integration coverage for tampered images, metadata, data objects, and relaxed mounts.
- Pin composefs to the upstream implementation and propagate errors from updated policy APIs.
| File | Description |
|---|---|
tmt/tests/tests.fmf |
Registers the verity integration test for image-mode hosts. |
tmt/plans/integration.fmf |
Adds a composefs-only verity test plan. |
crates/tests-integration/src/booted/composefs_verity.rs |
Tests integrity enforcement using disposable repositories. |
crates/tests-integration/src/booted.rs |
Registers the composefs-verity subcommand. |
crates/tests-integration/README.md |
Documents suite usage, coverage, and isolation. |
crates/tests-integration/Cargo.toml |
Adds dependencies for exercising the mount helper. |
crates/lib/src/store/mod.rs |
Propagates errors when relaxing repository policy. |
crates/lib/src/bootc_composefs/repo.rs |
Updates repository policy calls to handle errors. |
crates/lib/src/bootc_composefs/boot.rs |
Handles policy-setting errors during boot setup. |
crates/initramfs/src/lib.rs |
Enforces digest-based mounting and adds rejection tests. |
crates/initramfs/Cargo.toml |
Adds temporary-directory support for unit tests. |
Cargo.toml |
Temporarily pins composefs to the upstream digest-mount implementation. |
Cargo.lock |
Records the pinned composefs source and dependency updates. |
Review effort: Balanced
| # Repository::mount_digest, until it is released. Replace with a bump of the | ||
| # composefs-ctl version above once that happens. | ||
| [patch.crates-io] | ||
| composefs-ctl = { git = "https://github.com/composefs/composefs-rs", rev = "b43c311b2c32ca3c1bb18c2c612df84ad5a96dcc" } |
There was a problem hiding this comment.
Fable output:
Yes. The latest push (c53c2fc) updates this: the temporary [patch.crates-io] pin is gone and the dependency is now composefs-ctl = "0.10.0" from crates.io. Thank you for releasing composefs-rs 0.10.0!
|
Also from the previous PR
This needs investigation for sure. If this is true...then what we may need to do is change the logic here to detect whether the commandline came from a location that has verification (UKIs, especially with secure boot) |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
A strict composefs command line must not inherit an insecure repository policy from meta.json. Verify the opened image against the requested digest and mount that same file descriptor with verity required. This preserves valid boots from older installations with unprotected metadata and also verifies images resolved through named refs. Add unit and TMT coverage for stripped image verity, metadata replacement, wrong image refs, tampered data objects, and explicitly relaxed mounts. The mount suite passes with the fix and rejects the original behavior. Generated-by: AI Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>
composefs-rs now provides Repository::mount_digest, which takes the expected digest as a typed value, only looks at images/<digest> and requires the measured fs-verity digest to match before mounting. That is the check the previous commit open-coded around open_image; use the library API instead so there is one implementation of it, shared with composefs-setup-root. composefs-rs also records whether a repository's fs-verity requirement was detected from meta.json or chosen by the caller, and an explicit choice is final for the handle. The initramfs sets `IntegrityPolicy::Explicit(Enabled)` for a strict `composefs=` digest and `Explicit(Disabled)` for one with the `?` marker, then mounts by digest either way. With the marker a missing fs-verity digest is tolerated, but one that can be measured still has to match. The name-based mount() no longer runs in the initramfs, so images/refs/ is never consulted for a boot, with or without the marker. An image that only exists under images/refs/<digest> is now reported as not found rather than as a digest mismatch, so the boot moves on to the next command line candidate the same way it does for a missing image. `Repository::set_insecure` now returns a `Result`, so the remaining callers propagate it. They all run on a freshly opened repository, where the detected policy may still be replaced, so their behavior is unchanged. The booted verity test lists each case's expected result in a table per mode, matches on the typed CompareVerityError::DigestMismatch rather than on the message, and gains a case where images/<digest> is repointed at another verified image, which both modes must reject as a digest mismatch. Bump composefs-ctl to 0.10.0, the release that carries these APIs (composefs/composefs-rs#439). Generated-by: AI Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>
5ce8120 to
c53c2fc
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
Filed #2632 related to this |
| name: &str, | ||
| allow_missing_fsverity: bool, | ||
| ) -> Result<OwnedFd> { | ||
| let digest = Sha512HashValue::from_hex(name).context("Parsing composefs digest")?; |
There was a problem hiding this comment.
There's also something I don't like here, we should be pushing up the requirement for name = digest earlier I think. Actually I think this code should be tightly coupled to the kernel cmdline parsing instead of us extracitng it.
But I know that's larger work that can be a followup
|
Summary of the compatibility investigation for this PR, reconciled with @cgwalters and @cgwalters-bot. Full per-tag evidence and logs: gist. Executive summaryThis change breaks one set of existing installs. We accepted that, and recovery is easy and tested. Affected: composefs installs made by bootc 1.16.0 through 1.16.14 with
Two independent VM runs reproduced this: @cgwalters-bot with 1.16.13 and 1.16.14 (cgwalters-devspace-sandbox#33), and ours with 1.16.0 and 1.16.14. Not affected:
btrfs is not affected: 1.16.x installs there also lack fs-verity on Recovery (boot-tested):
Still to do:
Follow-ups:
Before upgrading: am I affected?Run as root ( args=$(tr ' ' '\n' </proc/cmdline | grep -E '^composefs(\.digest)?=') || { echo "unaffected: not booted with composefs"; exit 0; }
for arg in $args; do
val=${arg#*=}
case $val in \?*) echo "unaffected: '?' set, fs-verity is optional"; exit 0;; esac
img=$(readlink -e /sysroot/composefs/images/${val##*:}) || continue
if lsattr "$img" 2>/dev/null | cut -d' ' -f1 | grep -q V; then echo "unaffected: strict, and the image has fs-verity"; exit 0; fi
echo "AFFECTED: strict $arg but $img has no fs-verity ($(findmnt -no FSTYPE /sysroot) root)"; exit 1
done
echo "unknown: no composefs image found for $args"; exit 2Results:
For ext4 alone, Recovery
The suggested initramfs error reads:
It is added only when the command line is strict and measuring fs-verity fails with "not supported by filesystem". DetailsWho has a strict command line without working fs-verity, boot tests, follow-up proposalsWho has a strict command line without working fs-verityFor BLS, bootc chooses the policy at install time from the filesystem type. It doesn't probe whether verity works:
Upgrades copy the booted setting (since 1.14.0), so it stays fixed after install. A UKI carries its policy in its embedded command line, set when the UKI is built.
Only 1.16.0, 1.16.14 and #2629's head were boot-tested; the other rows come from the source. On ext4 with verity, 1.16.x installs still got fs-verity on their objects, so they keep booting after #2629. Boot testsSetup:
Not tested:
On legacy-BIOS GRUB, #2629's bootc can't upgrade at all ("SystemNotUEFI"). That's #2605 (proposed fix in #2602), unrelated to this change. Follow-ups (proposals, not part of #2629)
Generated-by: https://github.com/jmarrero-bot#llms |
|
btrfs is not affected. 1.16.x installs on btrfs also get a strict
* This is separate from this change. Composefs I've updated the check script in the summary above: the first version could wrongly print AFFECTED on a 1.16.0 system already switched to a UKI built by this PR. Details and logs: gist comment. Generated-by: https://github.com/jmarrero-bot#llms |
`just test-upgrade` starts from whatever bootc the published base image ships that day, so it never covers an upgrade from a release we know users run, such as the 1.16.x behind bootc-dev#2629. This builds a given release (a tag, or the latest one) from its own sources onto the current base image, installs it with that release's bootc via bcvk to-disk, then reuses the readonly plan's upgrade step to move to the local build. The release is built with its own install-buildroot and build-rpm, which have kept their interface since 1.15, rather than with its own Dockerfile: that fetches things which have since moved, and shares a cargo cache mount that can hand it another version's build. Before upgrading, the readonly plan now also checks that the release under test runs the system, records which build it was, and on composefs runs the pre-upgrade check from bootc-dev#2629, which must find a to-disk install unaffected. The composefs smoke test checks loader.conf only on installs by bootc 1.17.4 or later, which take it from the image. Part of bootc-dev#2632. Generated-by: AI
bootc 1.16.0 through 1.16.14 install with `to-filesystem` onto ext4 made without fs-verity, with a strict composefs argument, and since the change in bootc-dev#2629 the upgraded deployment fails to boot. A tmt plan can't wait for a boot that fails, so `--installer=to-filesystem` runs that VM itself: bcvk creates it, its disk is replaced by a to-filesystem install onto a stock mkfs.ext4, and the in-VM steps in pinned-to-filesystem.nu run over SSH. For 1.16.x this checks that the pre-upgrade check says AFFECTED, that the upgraded boot stops in bootc-root-setup with the expected error on the console, and both documented recoveries: booting the previous entry and then adding '?' to the new one (which the next upgrade keeps), and adding '?' before upgrading. Other releases must refuse the install. The expected console lines are one constant, to tighten once the clearer initramfs error lands. Part of bootc-dev#2632. Generated-by: AI
Each job is one `just test-upgrade-pinned` row, picked from the cases of the bootc-dev#2629 investigation (listed in the workflow's header): ostree and composefs to-disk installs from 1.15.2, the first and last 1.16 and the latest release, and the to-filesystem rows. A full run should take about a dozen runner-hours, so it runs weekly, on demand, and on PRs labeled ci/upgrade-pinned rather than on every PR. The installer axis is where image-builder and Anaconda installs are meant to go. Part of bootc-dev#2632. Generated-by: AI
On the composefs backend, the initramfs let the repository's
meta.jsondecide whether fs-verity was enforced.meta.jsonis an unsigned file on the same filesystem as the images, so a strictcomposefs=<digest>command line could end up mounting an image without fs-verity, and an image underimages/refs/<digest>could stand in for the pinned digest.With this change the kernel command line decides, not the repository:
composefs=<digest>setsIntegrityPolicy::Explicit(Enabled); one with the?marker setsExplicit(Disabled). An explicit policy is final for the repository handle.Repository::mount_digest, which only looks atimages/<digest>and requires the measured fs-verity digest to match.images/refs/is never consulted for a boot, with or without the marker.?, missing fs-verity is tolerated, but a digest that can be measured still has to match.meta.jsonlacks fs-verity while their image and objects carry it keep booting.Repository::set_insecurenow returns aResult, so the remaining callers propagate it; their behavior is unchanged.This bumps
composefs-ctlto 0.10.0, the release that providesmount_digestand the integrity policy (composefs/composefs-rs#439).Tests
images/refs/is not found in either mode. It runs unprivileged on tmpfs.composefs-verityTMT test, for the composefs backend only, builds disposable repositories and checks strict and relaxed command lines againstmeta.jsonwith and without fs-verity. The cases are: an untampered image, an image stripped of fs-verity, image andmeta.jsonboth stripped, a tampered data object,images/<digest>repointed at another verified image, and an image only present underimages/refs/. It skips itself when/var/tmpdoes not support fs-verity.Compatibility
A system with a strict command line whose image has no fs-verity now stops in
bootc-root-setupwhen booting the new deployment ("fs-verity is not supported by filesystem"). Such a system booted before only becausemeta.jsonturned enforcement off.From reading the code at the release tags, bootc could only produce that combination when all of these hold:
install to-filesystemorinstall to-existing-root;-O verity.A boot test of that case is reported in #2629 (comment).
Not affected: the ostree backend, xfs installs (they carry the
?marker since v1.14.0), andinstall to-diskon ext4 (bootc creates it with-O verity).To recover, add
?to the BLS entry, or rebuild the UKI withbootc container ukify --allow-missing-verity. The previous deployment carries its own initramfs and is expected to still boot when selected by hand; that has not been tested yet.Which installs historically got a strict command line, including ones made by other installers, is still being discussed in the comments below.
Generated-by: AI