fix(e2e): align container engine selection across helpers #1481

Activity

1. elezar

   added

   test:e2eRequires end-to-end coverageRequires end-to-end coverage

   area:buildRelated to CI/CD and buildsRelated to CI/CD and builds

   topic:testing

   on May 20, 2026
2. TaylorMutch commented on May 20, 2026

   

   TaylorMutch

   on May 20, 2026

   Collaborator

   More actions

   I've run into this as well. Thank you for the issue!
3. jganoff commented on May 21, 2026

   

   jganoff

   on May 21, 2026

   Contributor

   More actions

   I just ran into this as well, and also with the Helm / Skaffold dev setup scripts. Including some Claude-gathered context below for reference, since it may be helpful to fix/align in 1 pass.

   ---

   This also bites outside the e2e harness, in the helm/Skaffold local dev path. Sharing the repro in case it tightens the fix scope.

   Repro

   macOS with Docker Desktop running and Podman installed but its machine not running:

   mise run helm:k3s:create
   mise run helm:skaffold:dev

   Output:

   [container-engine] auto-detected: podman (override with CONTAINER_ENGINE=docker|podman)
   Staging prebuilt Rust binaries for Docker target 'gateway'...
   unsupported architecture: OS: darwin/arm64
   

   Workaround: CONTAINER_ENGINE=docker mise run helm:skaffold:dev.

   Why this case adds something to the precedence design

   The order proposed in the issue body would fix the inconsistency between the Rust e2e harness and container-engine.sh. The helm/Skaffold path has a stronger constraint worth encoding too:

   The engine used to build an image must match the engine the destination cluster runs on.

   Skaffold names it itself in the build log: Found [k3d-openshell-dev-...] context, using local docker daemon. k3d ran on Docker; a podman-built image is not visible to that daemon. Had podman info returned a valid arch, the build would have succeeded and deploy would have failed with ImagePullBackOff instead. Today's loud build failure is the milder symptom of the same root issue.

   I'd add a step to the precedence list:

   4. derive from kubectl config current-context when a kube context is the build target (e.g. k3d-* → docker, kind-on-podman → podman). Below explicit env overrides, above bare host auto-detect.

   Adjacent bug, separately filable

   ce_info_arch (tasks/scripts/container-engine.sh:176-182) suppresses podman info errors with 2>/dev/null and returns whatever lands on stdout. With a non-running podman machine it returned OS: darwin/arm64, which then flowed through normalize_arch → PREBUILT_ARCH → target_triple and finally surfaced as unsupported architecture: OS: darwin/arm64 from stage-prebuilt-binaries.sh four scripts later. Validating the arch string inside ce_info_arch would turn this into a clear "is your container engine running?" error.
4. elezar commented on May 22, 2026

   

   elezar

   on May 22, 2026

   MemberAuthor

   More actions

   🏗️ build-plan

   Implementation Plan

   Issue type: fix
   Complexity: Medium
   Confidence: High - the resolver policy has been narrowed; implementation is cross-file but well scoped

   Summary

   Align container engine selection across e2e helpers, shell build helpers, and Skaffold local image builds. The fix removes the duplicate OPENSHELL_E2E_CONTAINER_ENGINE selector, makes CONTAINER_ENGINE=docker|podman the single explicit engine override, and adds a scoped CONTAINER_ENGINE_TARGET=local-k8s-cluster hint for Skaffold push: false local-cluster image workflows.

   Scope

   • tasks/scripts/container-engine.sh: implement the canonical shell resolver, validate env values, reject removed OPENSHELL_E2E_CONTAINER_ENGINE, reject explicit conflicts, infer local-cluster engines only when CONTAINER_ENGINE_TARGET=local-k8s-cluster, and harden ce_info_arch error handling.
   • deploy/helm/openshell/skaffold.yaml: set CONTAINER_ENGINE_TARGET=local-k8s-cluster around the gateway and supervisor custom image build commands.
   • e2e/rust/src/harness/container.rs: align the Rust e2e helper with the e2e-relevant env rules only: CONTAINER_ENGINE, OPENSHELL_E2E_DRIVER, removed-selector failure, and conflict errors. Do not add kube-context inference in Rust.
   • e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh: validate CONTAINER_ENGINE against the lane, set it after validation, and remove OPENSHELL_E2E_CONTAINER_ENGINE exports.
   • architecture/build.md: document the final precedence model, the local-cluster target hint, and the Skaffold/local image constraint.
   • Published docs: update only if an existing user-facing Helm/Skaffold page already documents the affected workflow.

   Implementation Steps

   1. Refactor tasks/scripts/container-engine.sh so selection follows this algorithm:
      • reject OPENSHELL_E2E_CONTAINER_ENGINE with a migration message;
      • validate CONTAINER_ENGINE_TARGET, accepting only local-k8s-cluster;
      • compute any required engine from OPENSHELL_E2E_DRIVER=docker|podman;
      • compute any required engine from the active local cluster only when CONTAINER_ENGINE_TARGET=local-k8s-cluster;
      • validate explicit CONTAINER_ENGINE=docker|podman against those required engines;
      • otherwise use the e2e-required engine, then the local-cluster-required engine, then host auto-detection.
   2. Implement local-cluster inference in the shell helper:
      • k3d-* contexts imply Docker;
      • kind-* uses KIND_EXPERIMENTAL_PROVIDER=podman when set, otherwise Docker;
      • unrecognized contexts require explicit CONTAINER_ENGINE=docker|podman rather than falling back silently.
   3. Harden ce_info_arch so malformed engine metadata, such as OS: darwin/arm64, produces a direct error instead of flowing into staging scripts.
   4. Update the Rust e2e helper to use CONTAINER_ENGINE and OPENSHELL_E2E_DRIVER consistently, with conflict errors, but keep local-cluster inference shell-only.
   5. Update the Docker and Podman e2e wrappers to reject conflicting CONTAINER_ENGINE values and remove all OPENSHELL_E2E_CONTAINER_ENGINE exports.
   6. Wire CONTAINER_ENGINE_TARGET=local-k8s-cluster into Skaffold custom build commands for gateway and supervisor images.
   7. Update architecture documentation with the env model and local image workflow constraint.

   Test Plan

   • Unit tests: Add Rust tests in e2e/rust/src/harness/container.rs for removed OPENSHELL_E2E_CONTAINER_ENGINE failures, CONTAINER_ENGINE validation, OPENSHELL_E2E_DRIVER derivation, Docker/Podman conflict errors, invalid env values, and fallback behavior.
   • Shell validation: Defer a reusable shell test harness to a follow-up issue. During implementation, validate tasks/scripts/container-engine.sh with focused fixture-based command checks covering explicit env precedence, invalid values, removed-selector failure, e2e driver conflicts, k3d/kind local-cluster inference, unknown local-cluster context failures, explicit local-cluster conflict failures, and malformed ce_info_arch output. Keep temporary fixture scripts out of the final diff unless a minimal permanent check can be added without introducing a harness.
   • E2E tests: Because e2e helper files change, run the relevant local lane, usually mise run e2e:docker on a Docker host and mise run e2e:podman when Podman is available. For the Helm/Skaffold path, run mise run helm:k3s:create followed by a focused mise run helm:skaffold:run, or document why the environment cannot run it.

   Risks & Open Questions

   • kind-* can be Docker- or Podman-backed. The first implementation should use KIND_EXPERIMENTAL_PROVIDER=podman when present and otherwise default to Docker, with explicit CONTAINER_ENGINE available for unusual setups.
   • Unknown local Kubernetes contexts should require explicit CONTAINER_ENGINE because host auto-detection may choose an engine whose images are invisible to the cluster.
   • CONTAINER_ENGINE_TARGET=local-k8s-cluster must not be applied globally. Normal image builds should not depend on the user’s active kube context.
   • Some scripts may rely on ce_info_arch falling back to uname -m; tightening this behavior should keep errors actionable.
   • LSM compatibility: not affected. This change does not touch process identity, /proc, binary execution policy, or inter-process visibility.

   Documentation Impact

   • Update architecture/build.md with the canonical env model and local-cluster image workflow behavior.
   • Published docs are only needed if an existing user-facing local Helm/Skaffold page documents these commands.

   ---

   Revision 1 - initial plan
5. elezar

   added

   state:review-readyReady for human reviewReady for human review

   on May 22, 2026
6. github-actions commented on Jun 14, 2026

   

   github-actionsBot

   on Jun 14, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
7. github-actions

   added

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Jun 14, 2026
8. TaylorMutch

   removed

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Jun 15, 2026
9. elezar

   added

   state:agent-readyApproved for agent implementationApproved for agent implementation

   state:in-progressWork is currently in progressWork is currently in progress

   state:review-readyReady for human reviewReady for human review

   and removed

   state:review-readyReady for human reviewReady for human review

   state:in-progressWork is currently in progressWork is currently in progress

   state:agent-readyApproved for agent implementationApproved for agent implementation

   on Jun 16, 2026
10. maxamillion

    closed this as completedin #1944on Jun 22, 2026