Keep Experiment scope launches working across updates

docs/guide.md

@@ -986,7 +986,7 @@ Run `sudo nginx -t`, then `sudo systemctl restart nginx` on a systemd host. The
 thirdshift update
 ```
 
-It replaces the installed binary with the latest stable GitHub Release, or says it is already on it. Messages go to stderr and stdout stays empty. It exits `0` when it updated or was already up to date, and `1` on any failure, such as no network. Updating is safe while a Ticket run or a Spec run is using the old binary: it keeps running it. So do the child Ticket runs it starts afterwards, a Ticket run or a Base fix, which it starts through `/proc/self/exe` from the binary it is running rather than from the install path, so they run its version even once that path holds the new one or nothing at all. A Ticket run never checks for updates or updates itself.
+It replaces the installed binary with the latest stable GitHub Release, or says it is already on it. Messages go to stderr and stdout stays empty. It exits `0` when it updated or was already up to date, and `1` on any failure, such as no network. Updating is safe while a Ticket run, Spec run or Experiment is using the old binary: it keeps running it. So do the child Ticket runs it starts afterwards, a Ticket run or a Base fix, which it starts through `/proc/self/exe` from the binary it is running rather than from the install path, so they run its version even once that path holds the new one or nothing at all. An Experiment's later Trials and Graders keep its original OOM wrapper through the owning process's `/proc/<pid>/exe` link, including scopes prepared before the update. The requested Arm programs, arguments, environment and memory caps stay the same. A Ticket run never checks for updates or updates itself.
 
 `thirdshift update` only replaces a copy put in place by the shell installer (the `curl` command above), which leaves an install receipt in `~/.config/thirdshift/`. It refuses to touch any other copy and lists the command that updates each other kind of install: `cargo install thirdshift` for a crates.io install, pulling and reinstalling for a build from source (see [Building from source](#building-from-source)), and the `curl` command for a copy placed by hand.
 

site/architecture/index.html

@@ -273,7 +273,7 @@ <h1>Architecture</h1>
       <p>The Bar also saves <code>min_discriminating_tasks</code>, defaulting to 20, before the first Trial. <code>--min-discriminating-tasks</code> sets it for one Experiment; a changed minimum refuses resume. A shorter fixed discriminating list makes an <strong>Advisory suite</strong>, still shown with its Verdict and intervals. Overall aggregation, the Void dropout guard and required rechecks use only suites that are not advisory. All suites advisory gives inconclusive. CLI text, JSON and the Console mark Advisory suites and show judged and measurement counts. The release summary lists their Verdicts and keeps asking on fail or inconclusive. Saved Experiments without the minimum keep their original Bar and Verdict.</p>
       <p>Trial integrity protects the issue text, declared inputs and commits under review. Review commits, local-origin pushes and report files are allowed. Review code Graders receive a private copy of the prepared repository captured before the session, so session edits cannot change the code its checks grade. Process code Graders export the final commit and replace the declared test files and shared fakes from the private reference fix before running the task’s selectors. After the Trials, the same export and overlay run each Arm's new test files on the base and on the other Arm's final commit; the results are kept per pair under the Experiment, apart from the Scores.</p>
       <p>Trial completion removes Harness caches from each attempt's Session home, retaining native transcripts and Session evidence. Once grading and all rechecks finish, the Experiment removes every attempt's repository copy. Paused and interrupted Experiments retain repositories for resume. Requests, results, traces, reports, Grader packets and Scores stay readable; pruning failures only warn.</p>
-      <p>Each Trial, code Grader and model Grader gets its own <code>systemd-run --user --scope</code> with <code>MemoryHigh</code>, <code>MemoryMax</code>, <code>MemorySwapMax</code> and <code>OOMPolicy=kill</code>. Its process raises <code>oom_score_adj</code> before starting work. The <a href="/docs/#user-config">User config</a> defaults are 1 GiB, 2 GiB and 1 GiB; scopes require a reachable user manager with lingering enabled for logged-out work.</p>
+      <p>Each Trial, code Grader and model Grader gets its own <code>systemd-run --user --scope</code> with <code>MemoryHigh</code>, <code>MemoryMax</code>, <code>MemorySwapMax</code> and <code>OOMPolicy=kill</code>. Its process raises <code>oom_score_adj</code> before starting work. The wrapper uses the owning Experiment process's <code>/proc/&lt;pid&gt;/exe</code> link, so Trials and Graders keep its original wrapper after an update replaces or removes the install path, including scopes prepared before the update. The requested Arm programs, arguments, environment and caps stay the same. The <a href="/docs/#user-config">User config</a> defaults are 1 GiB, 2 GiB and 1 GiB; scopes require a reachable user manager with lingering enabled for logged-out work.</p>
       <p>Ordinary Harness sessions use uncapped scopes with <code>OOMPolicy=continue</code>: a child tool's kernel OOM kill leaves the Harness running so it can observe the tool's failure and recover. If the Harness itself fails, its cause names the Unix signal when present and adds <code>(scope OOM-killed)</code> when scope evidence confirms an OOM kill.</p>
       <p>The Experiment owns recheck history beside its immutable Grader Scores. <code>thirdshift experiments <wbr>&lt;id&gt; --json</code> exposes pending borderline and decisive Scores, naming each decisive flip and its resulting provisional Verdict. <code>thirdshift recheck <wbr>&lt;id&gt; &lt;trial&gt; pass|fail --reason '…'</code> takes the Experiment's exclusive lock and atomically saves the history, recalculated Verdict and pending list. <code>thirdshift recheck <wbr>&lt;id&gt; --pending</code> lists the pending Scores and their fingerprint; <code>thirdshift recheck <wbr>&lt;id&gt; --batch &lt;fingerprint&gt; pass|fail --reason '…'</code> checks that list under the same lock and records every listed Score in one atomic save. A stale fingerprint changes nothing. Rechecks stay in the CLI. They also update the latest invocation's Command outcome and Inbox group, preserving its completion time. The Verdict waits until the list is empty; flipping either pass/fail or a wrong-finding flag can make a Score decisive. Waiting rechecks place that invocation in Needs you; a confirmed pass moves it to Done.</p>
       <p><code>scripts/release.sh</code> runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.</p>

site/docs/index.html

@@ -1120,7 +1120,7 @@
 <div class="manual-content" id="updating">
 <pre><code class="language-sh">thirdshift update
 </code></pre>
-<p>It replaces the installed binary with the latest stable GitHub Release, or says it is already on it. Messages go to stderr and stdout stays empty. It exits <code>0</code> when it updated or was already up to date, and <code>1</code> on any failure, such as no network. Updating is safe while a Ticket run or a Spec run is using the old binary: it keeps running it. So do the child Ticket runs it starts afterwards, a Ticket run or a Base fix, which it starts through <code>/proc/self/exe</code> from the binary it is running rather than from the install path, so they run its version even once that path holds the new one or nothing at all. A Ticket run never checks for updates or updates itself.</p>
+<p>It replaces the installed binary with the latest stable GitHub Release, or says it is already on it. Messages go to stderr and stdout stays empty. It exits <code>0</code> when it updated or was already up to date, and <code>1</code> on any failure, such as no network. Updating is safe while a Ticket run, Spec run or Experiment is using the old binary: it keeps running it. So do the child Ticket runs it starts afterwards, a Ticket run or a Base fix, which it starts through <code>/proc/self/exe</code> from the binary it is running rather than from the install path, so they run its version even once that path holds the new one or nothing at all. An Experiment&#39;s later Trials and Graders keep its original OOM wrapper through the owning process&#39;s <code>/proc/&lt;pid&gt;/exe</code> link, including scopes prepared before the update. The requested Arm programs, arguments, environment and memory caps stay the same. A Ticket run never checks for updates or updates itself.</p>
 <p><code>thirdshift update</code> only replaces a copy put in place by the shell installer (the <code>curl</code> command above), which leaves an install receipt in <code>~/.config/thirdshift/</code>. It refuses to touch any other copy and lists the command that updates each other kind of install: <code>cargo install thirdshift</code> for a crates.io install, pulling and reinstalling for a build from source (see <a href="#building-from-source">Building from source</a>), and the <code>curl</code> command for a copy placed by hand.</p>
 <p>Keep a single copy: with copies in more than one directory on your <code>PATH</code> (say <code>~/.local/bin</code> and <code>~/.cargo/bin</code>), you can end up running a stale one without noticing. <code>type -a thirdshift</code> lists every copy on your <code>PATH</code>.</p>
 </div></details>

src/scope.rs

@@ -779,7 +779,7 @@ fn launch<S: AsRef<std::ffi::OsStr>>(
     }
     let mut command = scope_base(&unit, Some(caps));
     command
-        .arg(current_exe()?)
+        .arg(own_executable())
         .arg("--oom-exec")
         .arg(program)
         .args(args);
@@ -794,8 +794,12 @@ fn launch<S: AsRef<std::ffi::OsStr>>(
     })
 }
 
-fn current_exe() -> Result<PathBuf> {
-    std::env::current_exe().context("could not find the running thirdshift binary")
+/// Keep the running wrapper version after an update replaces or removes its
+/// install path (#1498). Another process executes this path, so `self` would
+/// name that process instead. Preserve the owning PID's proc link without
+/// resolving it to the possibly deleted install path.
+fn own_executable() -> PathBuf {
+    PathBuf::from(format!("/proc/{}/exe", std::process::id()))
 }
 
 /// Raise this process's own `oom_score_adj` to [`OOM_SCORE_ADJ`], best

src/update.rs

@@ -5,7 +5,9 @@
 //! release's installer moves the new binary into place with a rename, so a Run
 //! or a Spec run still using the old binary keeps running it. So do the child
 //! Runs it starts afterwards, which the child Run module starts from the
-//! running binary rather than from its install path.
+//! running binary rather than from its install path. An Experiment's later
+//! Trials and Graders also keep its original OOM wrapper through the owning
+//! process's `/proc/<pid>/exe` link, even for scopes prepared before the update.
 
 use std::fmt::Display;
 

tests/experiments.rs

@@ -66,6 +66,9 @@ mod pairs;
 #[path = "experiments/panel.rs"]
 mod panel;
 
+#[path = "experiments/update.rs"]
+mod update;
+
 #[test]
 fn saved_single_level_experiments_keep_their_bar_and_verdict_when_read_and_resumed() {
     let fixture = Fixture::new();

tests/experiments/update.rs

@@ -0,0 +1,72 @@
+use super::*;
+
+#[test]
+fn trials_and_graders_keep_running_after_the_install_path_is_replaced() {
+    survives_install_change(false, false);
+}
+
+#[test]
+fn trials_and_graders_keep_running_after_the_install_path_is_removed() {
+    survives_install_change(true, false);
+}
+
+#[test]
+fn a_scope_prepared_before_replacement_executes_the_original_wrapper() {
+    survives_install_change(false, true);
+}
+
+fn survives_install_change(remove: bool, prepared: bool) {
+    let fixture = Fixture::with_session("spec-review", 3);
+    let scenario = &fixture.scenario;
+    let wait = scenario.waits_to_be_replaced();
+    if prepared {
+        // Hold the first launch at the external systemd seam, after the
+        // Experiment has prepared its wrapper path and before it is exec'd.
+        support::write_executable(
+            &scenario.path("bin/systemd-run"),
+            &format!(
+                "#!/bin/sh\n{wait}\nwhile [ \"$#\" -gt 0 ] && [ \"$1\" != -- ]; do shift; done\nshift\nexec \"$@\"\n"
+            ),
+        );
+    }
+    scenario.agent_does(&format!("{}{wait}", review_reports_prelude()));
+    let replacement_ran = scenario.path("replacement-ran");
+    let result = scenario.run_copy_replaced_midway_with_env(
+        &Fixture::experiment_args(&[]),
+        &NO_SANDBOX,
+        |installed| {
+            if remove {
+                fs::remove_file(installed).unwrap();
+            } else {
+                let replacement = installed.with_extension("new");
+                support::write_executable(
+                    &replacement,
+                    &format!("#!/bin/sh\ntouch {}\nexit 1\n", replacement_ran.display()),
+                );
+                fs::rename(replacement, installed).unwrap();
+            }
+        },
+    );
+
+    assert_eq!(result.code, Some(0), "{}", result.stderr);
+    assert!(!replacement_ran.exists(), "the replacement wrapper ran");
+    let experiment: Value = serde_json::from_str(&result.stdout).unwrap();
+    let results = experiment["results"].as_array().unwrap();
+    assert_eq!(results.len(), 6, "later Trials must still dispatch");
+    for result in results {
+        assert_eq!(result["outcome"], "succeeded", "{result}");
+        assert_eq!(result["score"]["passed"], true, "{result}");
+    }
+    assert_eq!(fixture.trial_calls().len(), 6);
+    assert!(!fixture.grader_calls().is_empty(), "Graders must run too");
+    for call in fixture.grader_calls() {
+        let directory = PathBuf::from(call["cwd"].as_str().unwrap());
+        assert_eq!(
+            fs::read_to_string(directory.join("scores.json.adj"))
+                .unwrap()
+                .trim(),
+            "1000",
+            "the original OOM wrapper must run"
+        );
+    }
+}

tests/support/mod.rs

@@ -655,11 +655,23 @@ impl Scenario {
         &self,
         args: &[&str],
         replace: impl FnOnce(&Path),
+    ) -> RunResult {
+        self.run_copy_replaced_midway_with_env(args, &[], replace)
+    }
+
+    /// [`Scenario::run_copy_replaced_midway`] with child-only environment overrides.
+    pub fn run_copy_replaced_midway_with_env(
+        &self,
+        args: &[&str],
+        env: &[(&str, &str)],
+        replace: impl FnOnce(&Path),
     ) -> RunResult {
         let copy = self.path("installed/thirdshift");
         fs::create_dir_all(copy.parent().unwrap()).unwrap();
         copy_executable(Path::new(env!("CARGO_BIN_EXE_thirdshift")), &copy);
-        let held = self.spawn_until_started(self.command_running(&copy, args), COPY_IN_USE);
+        let mut command = self.command_running(&copy, args);
+        command.envs(env.iter().copied());
+        let held = self.spawn_until_started(command, COPY_IN_USE);
         replace(&copy);
         fs::write(self.path(COPY_REPLACED), "").unwrap();
         held.finish()