Skip to content

Keep Experiment scope launches working across updates - #1985

Merged
JacobStephens2 merged 2 commits into
mainfrom
issue-1498
Nov 10, 2026
Merged

JacobStephens2 merged 2 commits into
mainfrom
issue-1498

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Keep an Experiment's original OOM wrapper available after an update replaces or removes the installed binary.

 Trial / Grader scope launch
- systemd-run … <resolved install path> --oom-exec <requested program> …
+ systemd-run … /proc/<owning thirdshift PID>/exe --oom-exec <requested program> …

Confirmed cause: std::env::current_exe() resolves to an install path ending in (deleted) after the running executable is unlinked. The proc link stays unresolved, so another process can execute the original wrapper even when the command was prepared before the update. Arm programs, arguments, environment, caps and OOM policy retain their existing behavior. The manual and architecture page now describe Experiment update safety.

Closes #1498

Test seams: Experiment CLI with a copied running executable and the existing pass-through systemd launcher; Trial and Grader execution across install-path replacement/removal, including a scope prepared before replacement.

Evidence

Execution evidence

  • Before: cargo test --test experiments update:: -- --nocapture — 0 passed, 3 failed. Replacement and removal stop the next Trial with Arm Trial protocol failed (exit status: 127): …/installed/thirdshift (deleted): No such file or directory; the prepared command executes the replacement and exits 1.
    After: the same command — 3 passed, 0 failed. Each Experiment finishes all six Trials and its Graders; the replacement never runs, and Graders retain oom_score_adj=1000.
  • A separate real systemd-run --user --scope probe reproduced Failed to find executable … (deleted): No such file or directory after both rename-over replacement and removal. /proc/<owning PID>/exe executed the original probe in both cases. No production agents or production Experiment were used.
  • npm run docs --prefix site, npm run docs:check --prefix site, cargo fmt --all -- --check, and git diff --check passed.
  • Affected checks: PLAYWRIGHT_BROWSERS_PATH=/tmp/thirdshift-1498-browsers CARGO_BUILD_JOBS=3 NEXTEST_TEST_THREADS=3 RUST_TEST_THREADS=3 scripts/affected-tests refs/remotes/origin/main passed: 68 site checks (98.33 seconds) and the full Rust suite through the cargo test fallback (3963.46 seconds wall time, including compilation; test-target compilation took 5m 17s). The initial scripts/affected-tests main selected unrelated changes because local main was 330 commits behind the review base and checked out in another worktree; it stopped at missing Go. The successful rerun used the requested remote main fixed point without moving that other worktree's branch.

Visual evidence

Documentation wording at matching desktop and mobile viewports:

Manual — desktop 1440×900

Before
After

Manual — mobile 390×844

Before
After

Architecture — desktop 1440×900

Before
After

Architecture — mobile 390×844

Before
After

Review coverage

Independent review against refs/remotes/origin/main (9712febb): Standards 0 findings, Spec 0 findings. Both reviewers read all eight changed files, including the generated manual directly. Unread changed files: none. Generated-output exclusions: none.

Final reports and file lists: .thirdshift-review-e95af0/standards.md, .thirdshift-review-e95af0/spec.md; counts in summary.json.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Revert restores the former launcher lookup; no stored data or protocol changes.

Blast Radius: scopes

The shared capped launcher serves Trials and code/model Graders. It requires the owning thirdshift process to remain alive, as it does while supervising their launches; existing caps and scope policies are preserved.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit dbdd894 into main Nov 10, 2026
16 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1498 branch November 10, 2026 23:27
This was referenced Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Updating thirdshift during an Experiment stops it: the Trial scope launcher execs the deleted install path

1 participant