refactor(compute): standardize lifecycle coordination across drivers #4115

Activity

1. elezar commented on Oct 2, 2026

   

   elezar

   on Oct 2, 2026

   MemberAuthor

   More actions

   Additional suspected Docker instance for the cross-driver lifecycle audit:

   • Failed job: docker-e2e / E2E (rust-docker).
   • Test: on_failure_policy_replaces_runtime_and_preserves_workspace (tests/sandbox_lifecycle.rs:1316). The test deliberately exits the initial workload with code 17, then expects the on-failure replacement to reach Ready with restart count 1 and preserve the workspace.
   • Failure: the replacement did not complete its policy-driven restart within 60 seconds. The final details showed Phase: Error, Restart count: 1, no last exit code, and Ready: False (ContainerExited). The lifecycle suite finished with 17 passed and 1 failed.

   Gateway timeline for sandbox 2fae61ab-b92a-4e2e-b103-21c705b5acd7 (e2e-25213-7), UTC on 2026-10-02:

   12:35:24.179  Original main-process exit report completes
   12:35:24.207  Original supervisor session ends
   12:35:24.811  Replacement supervisor session accepted (new instance ID)
   12:35:24.823  Replacement policy reported loaded
   12:35:24.832  Replacement ReportEndpointStatus completes successfully
   12:35:24.995  Sandbox runtime restarted; waiting for replacement supervisor
   12:35:24.999  driver_watch: Sandbox phase changed Ready -> Error
   12:35:24.999  Sandbox failed to become ready: ContainerExited Container exited
   12:36:22.500  Test times out; sandbox remains Error
   12:36:22.565  Replacement supervisor session closes during test cleanup
   

   This resembles the restart/reconciliation race described in #3776 and addressed for Podman by #4117, but the mechanism is not confirmed for Docker. The available job logs do not distinguish a stale published exit snapshot from actual premature workload containment. The replacement supervisor connection alone does not establish that the workload stayed healthy.

   Suggested audit target: Docker lifecycle/watch/admission overlap during partial restart, including both protection against destructive reconciliation and filtering of stale state publication. A deterministic overlap reproduction with container-state and containment evidence is still needed before treating this as the same root cause or claiming a Docker fix.
2. aetherxeg-source commented on Oct 2, 2026

   

   aetherxeg-source

   on Oct 2, 2026

   Last edited by aetherxeg-source

   More actions

   The cross-driver contract should preserve:

   "resource identity != lifecycle generation"

   A driver can correctly resolve the logical sandbox and still act on the wrong incarnation of that sandbox.

   I would standardize lifecycle evidence around:

   "{sandbox_identity, execution_generation, operation_generation, authority}"

   and require destructive actions to carry an expected execution generation.

   A cross-driver conformance test could:

   1. create execution A;
   2. initiate delayed cleanup/observation from A;
   3. replace A with execution B under the same sandbox identity;
   4. deliver A's delayed action;
   5. verify every driver rejects it as stale without affecting B.

   Critically, this should be tested independently for observation filtering and destructive remote effects, because filtering a stale observation after a driver has already stopped B is too late.

   The core invariant is:

   "stale generation has zero destructive authority over a newer runtime"
3. elezar commented on Oct 6, 2026

   

   elezar

   on Oct 6, 2026

   MemberAuthor

   More actions

   Additional context from reviewing #4235 at 0cfcd1e193e8c486114c9e7d1ee75ea75a0cf2fa (fixes #4129):

   Docker evidence now distinguishes stale publication from containment

   This updates the earlier suspected Docker instance in this issue. #4129 documents a controlled reproduction that delays an authentic old-runtime exit snapshot until after replacement readiness: the gateway regresses Ready -> Error / ContainerExited while both replacement containers remain running. That establishes stale state publication as a separate failure mechanism; it does not establish that the original CI failure had only this cause, since #4235 also removes independent test timing assumptions.

   #4235 extends gateway live-state revalidation from Starting to Ready observations reporting ContainerExited, ContainerStopped, or ContainerRuntimeRestart. It waits for the local sandbox lifecycle gate and rereads driver/store state. Its deterministic coverage checks both stale and genuine terminal observations for all three reasons. Driver-side filtering alone cannot retract a snapshot already queued at the gateway. Gateway revalidation.

   Concrete remaining audit targets

   • Podman: emits the same three reasons and therefore benefits from fix(compute): revalidate late exit events after restart #4235. Its exit timestamp fences operate before publication, so already-queued snapshots still need gateway protection. This is distinct from the destructive reconciliation overlap addressed by fix(podman): serialize restart and workload containment #4117. Watcher/fences.
   • VM stale publication: ProcessExited, ProcessPollFailed, and ComputeStopped bypass the new Ready revalidation predicate. The gateway classifies Ready=False / ProcessExited as terminal, so a delayed old-runtime snapshot can still fail a ready replacement at the state-application boundary. This is a source-level gap, not a live-confirmed reproduction. Condition predicate, VM monitor.
   • VM cleanup ownership: the monitor captures a process handle, then later writes the canonical exit marker, updates the registry snapshot, invokes lifecycle extension cleanup, and releases GPU resources using the sandbox ID. Those later actions do not recheck that the registry still belongs to the observed process. Audit an old monitor paused after exit detection across stop/start and replacement publication. Gateway revalidation cannot repair a driver record already polluted by stale monitoring or undo destructive cleanup. This overlap remains unqualified. Monitor and status update.
   • Kubernetes: controller conditions such as Suspended=True / PodTerminated do not enter the new Ready revalidation path. Existing observed-generation filtering and bootstrap completion coordination are useful, but do not prove immunity to a converted old snapshot already queued downstream. Force that delivery boundary separately from controller/driver reconciliation. Bootstrap completion coordination.
   • Deleted events: these take a separate gateway path from snapshot revalidation. Audit whether a delayed removal observation can target a replacement under a reused sandbox identity. Treat this as an audit target, not a confirmed failure. Deleted-event application.

   Suggested deterministic coverage

   For each driver, hold an observation or cleanup action from runtime A, replace A with B under the same sandbox identity, then release A's action after B reaches Ready. Independently verify:

   1. Gateway phase remains Ready, exec succeeds, and workspace markers survive delayed old snapshots/deletion observations.
   2. Old cleanup cannot modify B's runtime resources, driver record, exit markers, extension-owned resources, or GPU ownership.
   3. Genuine B workload failure and supervisor loss remain observable and enforce required containment.
   4. Read failures during revalidation do not silently lose genuine terminal state indefinitely; exercise retry/reconciliation behavior as well as the successful-read path.

   The contract should cover every observation/cleanup path rather than infer lifecycle semantics solely from container-specific reason strings. Runtime-generation ownership and cross-process/HA/cancellation fencing remain separate work from an instance-local gate.

   No blocker identified in #4235 for its demonstrated bug, and it does not block this follow-up work: it changes no driver API or general lifecycle architecture. Source review and existing CI results support the observations above; this review did not independently repeat live qualification in other drivers.
4. aetherxeg-source commented on Oct 7, 2026

   

   aetherxeg-source

   on Oct 7, 2026

   More actions

   This decomposition is useful. I think it makes one remaining contract boundary especially important:

   stale-publication fencing != stale-mutation fencing

   The gateway revalidation in #4235 can reject or repair a stale terminal observation before it becomes authoritative state, but it cannot retroactively undo a driver-side cleanup or resource mutation that was already committed by runtime A after runtime B became authoritative.

   For the VM path in particular, I would make each post-observation side effect independently generation-bound:

   "observed runtime generation"
   "-> registry mutation"
   "-> exit-marker write"
   "-> extension cleanup"
   "-> GPU/resource release"

   A deterministic adversarial test can pause runtime A after exit detection but immediately before each one of those effect boundaries, publish and ready B, then release A.

   The required result should be:

   "A may complete bookkeeping about A, but A has zero authority to mutate B-owned state or resources."

   That also gives a cleaner cross-driver conformance rule than enumerating terminal reason strings: every delayed observation and every delayed destructive effect must prove ownership of the runtime generation it is about to modify.

   I would keep publication fencing and effect fencing as separate test families, because passing the former does not imply the latter.
5. github-actions commented on Oct 22, 2026

   

   github-actionsBot

   on Oct 22, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 14 days and is now marked stale. It may be closed in 14 days if there is no further activity. Comment or remove the state:stale label to keep it open.
6. github-actions

   added

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Oct 22, 2026