Skip to content

fix(dstack-util): re-initialize a data disk whose initialization was interrupted - #1339

Merged
kvinwang merged 1 commit into
nextfrom
fix/luks-half-initialized-disk
Sep 24, 2026
Merged

kvinwang merged 1 commit into
nextfrom
fix/luks-half-initialized-disk

Conversation

@kvinwang

@kvinwang kvinwang commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

A first boot interrupted between luksFormat and mkfs leaves a LUKS header with no filesystem behind it. Every later boot fails to mount the data disk and the CVM reboots forever.

Change

  • luksFormat sets the LUKS2 label dstack-initializing; it is cleared (cryptsetup config --label=) once the filesystem exists.
  • On boot, an encrypted disk whose header still carries the label is initialized again. Any other disk is mounted as before, so a damaged but recoverable filesystem is never reformatted.
  • The header validator accepts exactly this label (any other non-empty label is still rejected) and reports it from the in-memory header it already validated, so there is no second read of the disk.
  • On that retry only, zpool labelclear -f drops a pool label left by an interruption after zpool create, which would otherwise block zpool create (ext4 already uses mkfs.ext4 -F). First-time initialization is unchanged.

Disks initialized by earlier builds have an empty label and are unaffected. The label is in the unauthenticated header, so a host can force a re-initialization, but it could already do so by erasing the header.

Verification

  • cargo test -p dstack-util (112 passed), fmt, clippy clean.
  • Headers produced by cryptsetup luksFormat --label dstack-initializing and after cryptsetup config --label= validate as Ok(true) / Ok(false).
  • On a loop device mapped through a detached header, mkfs.ext4 then cryptsetup config --label= succeeds, the label is gone, and the ext4 behind it is intact.

@kvinwang
kvinwang force-pushed the fix/luks-half-initialized-disk branch 2 times, most recently from ed1637d to f43d4df Compare September 24, 2026 15:08
@kvinwang kvinwang changed the title fix(dstack-util): re-initialize a data disk whose LUKS header has no filesystem fix(dstack-util): re-initialize a data disk whose initialization was interrupted Sep 24, 2026
…interrupted

A first boot interrupted between luksFormat and mkfs left a LUKS header
with no filesystem behind it, so every later boot failed to mount the data
disk and the CVM rebooted forever.

luksFormat now sets the LUKS2 label `dstack-initializing`, and the label is
cleared once the filesystem exists. A disk that still carries the label is
initialized again; any other disk is mounted as before, so a damaged but
recoverable filesystem is never reformatted. The header validator accepts
this one label and reports it from the header it already validated.
@kvinwang
kvinwang force-pushed the fix/luks-half-initialized-disk branch from f43d4df to 1181b71 Compare September 24, 2026 15:26
@kvinwang
kvinwang merged commit cab5f31 into next Sep 24, 2026
11 checks passed
@kvinwang
kvinwang deleted the fix/luks-half-initialized-disk branch September 24, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant