Skip to content

bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug #1762

Closed as not planned

Description

@zakkak

Agent Diagnostic

Investigated the ProvisioningFailed error when creating sandboxes from large local images.

  • Root cause traced to crates/openshell-driver-vm/src/rootfs.rs:format_ext4_image_from_dir which calls mke2fs -d to populate an ext4 image from a staging directory.
  • The staging directory contains source-rootfs.tar (~4 GiB) copied in by stage_guest_image_payload at crates/openshell-driver-vm/src/driver.rs:4094-4104.
  • mke2fs -d in e2fsprogs 1.47.3 has a signed 32-bit overflow bug: any file with >2 GiB of non-zero data fails with __populate_fs: Ext2 file too big. Zero-filled files pass because mke2fs skips zero blocks as sparse.
  • Confirmed the 2 GiB boundary empirically (2048 MiB succeeds, 2049 MiB fails). Block size, filesystem features, and image size have no effect.
  • Additionally, e2fs_tool_candidates in rootfs.rs only searches macOS Homebrew paths (/opt/homebrew/opt/e2fsprogs, /usr/local/opt/e2fsprogs) after the bare tool name. On Linux, when the driver process runs with a restricted $PATH (e.g. missing /usr/sbin), the bare name lookup fails and no Linux-standard paths are tried, resulting in a misleading "not found" error.
  • Upstream bug report draft is at /tmp/mke2fs-bug-report.md for filing against tytso/e2fsprogs.

Description

Actual behavior: openshell sandbox create --from localhost/<image>:latest fails when using the VM compute driver with microVMs:

ProvisioningFailed: failed to create ext4 rootfs image from .../.openshell-overlay-staging-...:
__populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system

On some Linux systems the initial error is instead:

/usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found. Install e2fsprogs (mke2fs/mkfs.ext4) and retry

This occurs because the tool search only checks macOS Homebrew paths and does not include standard Linux paths like /usr/sbin, /sbin, or /usr/bin.

Expected behavior: Sandbox creation succeeds for any valid container image regardless of rootfs tar size, and mke2fs/mkfs.ext4 is found on standard Linux installations.

This only affects the VM compute driver (compute_drivers = ["vm"]), which uses mke2fs -d to build ext4 rootfs images. The Podman and Docker drivers are not affected.

Reproduction Steps

  1. Configure the gateway to use the VM compute driver (see gateway.toml below).
  2. Build or pull a local container image whose exported rootfs tar exceeds 2 GiB of non-zero content.
  3. Run:
    openshell sandbox create \
      --name test \
      --cpu 2 \
      --memory 2Gi \
      --from localhost/<image>:latest
  4. Observe the ProvisioningFailed error.

Environment

  • OS: Fedora 44 (Linux 7.0.10-201.fc44.x86_64)
  • e2fsprogs: 1.47.3
  • OpenShell: 0.0.55 / 0.0.56-dev

gateway.toml (using microVMs via the VM compute driver):

[openshell]
version = 1

[openshell.gateway]
bind_address = "0.0.0.0:17670"
compute_drivers = ["vm"]

[openshell.drivers.vm]
vcpus = 8
mem_mib = 16384

Logs

Adding the output after applying the following patch


diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs
index 904ed8cd..cb3d222d 100644
--- a/crates/openshell-driver-vm/src/rootfs.rs
+++ b/crates/openshell-driver-vm/src/rootfs.rs
@@ -458,7 +458,7 @@ fn round_up_to_mib(bytes: u64) -> u64 {
 }
 
 fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), String> {
-    let mut last_error = None;
+    let mut errors = Vec::new();
     for tool in ["mke2fs", "mkfs.ext4"] {
         for candidate in e2fs_tool_candidates(tool) {
             let label = candidate.display().to_string();
@@ -467,6 +467,8 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
                 .arg("-F")
                 .arg("-t")
                 .arg("ext4")
+                .arg("-b")
+                .arg("4096")
                 .arg("-E")
                 .arg("root_owner=0:0")
                 .arg("-d")
@@ -476,7 +478,7 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
             match output {
                 Ok(output) if output.status.success() => return Ok(()),
                 Ok(output) => {
-                    last_error = Some(format!(
+                    errors.push(format!(
                         "{label} failed with status {}\nstdout: {}\nstderr: {}",
                         output.status,
                         String::from_utf8_lossy(&output.stdout),
@@ -484,18 +486,18 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St
                     ));
                 }
                 Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
-                    last_error = Some(format!("{label} not found"));
+                    errors.push(format!("{label} not found"));
                 }
                 Err(err) => {
-                    last_error = Some(format!("run {label}: {err}"));
+                    errors.push(format!("run {label}: {err} (kind: {:?})", err.kind()));
                 }
             }
         }
     }
     Err(format!(
-        "failed to create ext4 rootfs image from {}: {}. Install e2fsprogs (mke2fs/mkfs.ext4) and retry",
+        "failed to create ext4 rootfs image from {}: [{}]. Install e2fsprogs (mke2fs/mkfs.ext4) and retry",
         source.display(),
-        last_error.unwrap_or_else(|| "no ext4 formatter found".to_string())
+        errors.join("; ")
     ))
 }
 
@@ -736,6 +738,9 @@ fn run_debugfs(image_path: &Path, command: &str) -> Result<(), String> {
 
 fn e2fs_tool_candidates(tool: &str) -> Vec<PathBuf> {
     let mut candidates = vec![PathBuf::from(tool)];
+    for dir in ["/usr/sbin", "/sbin", "/usr/bin"] {
+        candidates.push(Path::new(dir).join(tool));
+    }
     for root in ["/opt/homebrew/opt/e2fsprogs", "/usr/local/opt/e2fsprogs"] {
         candidates.push(Path::new(root).join("sbin").join(tool));
         candidates.push(Path::new(root).join("bin").join(tool));


Output


✗ Error: ProvisioningFailed: failed to create ext4 rootfs image from /home/zakkak/.local/state/openshell/vm-driver/images/sandbox-prepared-rootfs-ext4-umoci-v3-openshell-0.0.56-dev.8-gb41e0df4-sha256-7f5f917046be984da8aa6821cff29b225138033bc0c0387730feca99c5e97b16.staging-1780583285334-0/.openshell-overlay-staging-357730-1780583291284: [mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /usr/sbin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /sbin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /usr/bin/mke2fs failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mke2fs: Ext2 file too big while populating file system
; /opt/homebrew/opt/e2fsprogs/sbin/mke2fs not found; /opt/homebrew/opt/e2fsprogs/bin/mke2fs not found; /usr/local/opt/e2fsprogs/sbin/mke2fs not found; /usr/local/opt/e2fsprogs/bin/mke2fs not found; mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /usr/sbin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /sbin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /usr/bin/mkfs.ext4 failed with status exit status: 1
stdout: 
stderr: __populate_fs: Ext2 file too big while writing file "source-rootfs.tar"
mkfs.ext4: Ext2 file too big while populating file system
; /opt/homebrew/opt/e2fsprogs/sbin/mkfs.ext4 not found; /opt/homebrew/opt/e2fsprogs/bin/mkfs.ext4 not found; /usr/local/opt/e2fsprogs/sbin/mkfs.ext4 not found; /usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found]. Install e2fsprogs (mke2fs/mkfs.ext4) and retry

Agent-First Checklist

  • I pointed my agent at the repo and had it investigate this issue
  • I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
  • My agent could not resolve this — the diagnostic above explains why

Activity

  1. added theissue type on Jun 4, 2026
  2. TaylorMutch commented on Jul 13, 2026

    @TaylorMutch
    Collaborator

    📋 triage-agent

    Triage Assessment

    Classification: bug-confirmed

    Summary

    This is a confirmed VM compute-driver compatibility defect, not user error and not a duplicate. Current main retains both reported failure modes: large local-image archives are passed through the affected mke2fs -d population path, and e2fs tool discovery can miss standard Linux installations when the gateway has a restricted PATH.

    Investigation

    stage_guest_image_payload still copies a local image export into the ext4 staging tree as one source-rootfs.tar, and format_ext4_image_from_dir still populates the image with system mke2fs/mkfs.ext4 -d. The reported large-file error matches the upstream e2fsprogs offset overflow fixed by tytso/e2fsprogs commit 6359e0ec and included in e2fsprogs 1.47.4; installations without that fix remain susceptible, including distributions that have not backported it.

    Independently, e2fs_tool_candidates still searches only the bare command and Homebrew paths. It does not try standard Linux locations such as /usr/sbin, and later missing candidates can replace a meaningful formatter error with a misleading final "not found" message. No related OpenShell issue or pull request was found.

    Recommendation

    Route this to the VM compute-driver owners as compatibility work. Remediation should cover Linux tool discovery and useful error reporting, account for fixed and unfixed e2fsprogs variants, and document relevant prerequisites; whether to retain or redesign the current image-construction pipeline is a separate implementation decision.

  3. github-actions commented on Aug 29, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions