bug: rust-vm E2E intermittently fails with masked ext4 formatter error #2423

Activity

1. pimlock

   added

   os:linuxIssue affects Linux hostsIssue affects Linux hosts

   topic:testing

   github_actionsPull requests that update GitHub Actions codePull requests that update GitHub Actions code

   on Jul 22, 2026
2. pimlock commented on Jul 22, 2026

   

   pimlock

   on Jul 22, 2026

   Last edited by pimlock

   CollaboratorAuthor

   More actions

   🏗️ build-plan

   Implementation Plan

   Issue type: fix
   Complexity: Medium
   Confidence: High — clear path; the masked formatter stderr prevents absolute confirmation of the disk-pressure trigger

   Summary

   Reduce the rust-vm job's peak temporary-disk usage by deleting downloaded and expanded registry-layer staging after the merged rootfs is complete and before the ext4 image is written. Validate the e2fsprogs tools already present on the GitHub-hosted runner without installing or changing packages, and preserve actionable errors from formatter candidates that actually executed instead of replacing them with later missing Homebrew probes.

   Scope

   • .github/workflows/e2e-test.yml: validate the preinstalled e2fsprogs tools required by the rust-vm E2E lane without modifying the runner's package set.
   • crates/openshell-driver-vm/src/driver.rs: reclaim staging_dir/layers after successful layer application and before rootfs preparation/ext4 formatting; propagate cleanup failures.
   • crates/openshell-driver-vm/src/rootfs.rs: prioritize and retain status/stdout/stderr from executable formatter candidates, with candidate-injected logic that is unit-testable without changing global PATH.

   Implementation Steps

   1. Validate that mke2fs, mkfs.ext4, and debugfs resolve on the vanilla ubuntu-24.04 host before running the E2E suite. Do not install e2fsprogs in this job.
   2. After registry layers have been successfully applied into the merged rootfs, remove the compressed and expanded layers/ staging tree before rootfs preparation and ext4 creation. Abort with an actionable error if cleanup fails.
   3. Refactor ext4 formatter candidate handling so fallback candidates are still attempted, but later NotFound results cannot overwrite failures from commands that ran. When no candidate executes, retain the existing installation guidance.
   4. Add focused unit coverage for staging cleanup and formatter-error selection, then run the driver tests and repository pre-commit suite.
   5. Run the VM E2E lane on an Ubuntu/KVM host when available and use repeated GitHub-hosted rust-vm runs to validate the intermittent reliability fix.

   Test Plan

   • Unit tests: In rootfs.rs, cover a failing executable formatter followed by missing candidates, all candidates missing, and a later successful fallback without relying on global PATH. In driver.rs, verify layer cleanup removes layers/ while preserving the merged rootfs/.
   • Integration tests: Run cargo test -p openshell-driver-vm and mise run pre-commit.
   • E2E tests: Run mise run --no-deps --skip-deps e2e:vm on Ubuntu/KVM if available. Confirm at least three consecutive GitHub-hosted E2E (rust-vm) executions after the change because the original failure is intermittent.

   Risks & Open Questions

   • Disk pressure is strongly indicated but the exact system-formatter stderr is irretrievable from past logs because it was overwritten. The failures take long enough to show that at least one formatter executed, all matching runs used the same image digest, and a separate rust-vm attempt explicitly exhausted disk.
   • Cleanup after all layers are applied removes the formatter-stage peak. If pull/apply itself later exhausts disk, deleting each blob after extraction and each layer root after application should be handled as a narrower follow-up.
   • Cleanup failure should abort rather than warn; continuing would preserve the known high-water mark and recreate the flaky behavior.
   • Validation-only intentionally relies on the current GitHub runner image. If a future image removes e2fsprogs, this job should fail immediately and provisioning should be addressed at the runner-image or shared host-setup level.
   • Keep this issue limited to ext4 formatter reporting. The similar debugfs last-error pattern is not implicated and can be addressed separately.

   Documentation Impact

   No gateway configuration, Helm rendering, published user workflow, or stable architecture boundary changes. The VM driver README already documents e2fsprogs as a prerequisite, so no documentation update is expected.

   LSM Compatibility

   No SELinux/AppArmor compatibility impact is expected. Cleanup is host-side inside the VM driver's own staging directory before guest launch, and tests will avoid fork/exec probes that depend on cross-label /proc/<pid>/exe access.

   ---

   Revision 2 — validate preinstalled e2fsprogs tools without installing packages
   Revision 1 — initial plan
3. pimlock

   added

   state:review-readyReady for human reviewReady for human review

   state:agent-readyApproved for agent implementationApproved for agent implementation

   state:in-progressWork is currently in progressWork is currently in progress

   state:pr-openedPR has been opened for this issuePR has been opened for this issue

   and removed

   state:review-readyReady for human reviewReady for human review

   state:in-progressWork is currently in progressWork is currently in progress

   on Jul 22, 2026
4. pimlock commented on Jul 22, 2026

   

   pimlock

   on Jul 22, 2026

   CollaboratorAuthor

   More actions

   Implemented in #2425.

   Summary:

   • Reclaim temporary compressed and extracted registry layers before formatting the merged VM rootfs, reducing peak runner disk usage without changing the durable image cache.
   • Preserve actual formatter execution failures instead of replacing them with a later missing fallback-path error.
   • Validate the GitHub-hosted rust-vm runner's existing mke2fs, mkfs.ext4, and debugfs tools without installing e2fsprogs in the job.

   Verification:

   • mise exec -- cargo test -p openshell-driver-vm — 135 tests passed (123 library, 12 binary).
   • mise run pre-commit — passed, including formatting, clippy, workspace tests, and server feature tests.
   • E2E phase not applicable under the build workflow because no files under e2e/ changed; PR CI will exercise the updated rust-vm workflow.

   Deviations from the approved plan: none.
5. drew

   closed this as completedin #2425on Jul 22, 2026
6. pull

   added a commit that references this issue on Jul 22, 2026

   fix(vm): reduce registry rootfs staging pressure (NVIDIA#2425)

   ...

   This issue will close once commit ca31805 is merged into the 'main' branch.

   ca31805