Skip to content

Give every Trial and Grader a memory-capped scope - #1225

Merged
JacobStephens2 merged 6 commits into
issue-977from
issue-1015
Oct 24, 2026
Merged

JacobStephens2 merged 6 commits into
issue-977from
issue-1015

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

Experiment
  Trial → own memory-capped scope → success, failure or memory_cap Void
  Code Grader → own memory-capped scope → mechanical Score
  Model Grader → own memory-capped scope → written Score

Every scope raises its own oom_score_adj and uses OOMPolicy=kill, so a tool's OOM cannot leave its Trial successful. Missing user-manager access stops dispatch; logged-out launchers discover the user's runtime directory. User config documents the 1G / 2G / 1G defaults, and repeated cap kills remain Arm findings.

Closes #1015

Evidence

Before: the independent reviewers' targeted reproductions failed: code Graders inherited OOM score 0; an unreachable manager dispatched uncapped Trials; a cap-killed tool left a succeeded, passed Trial; 2GG was accepted; and both defaults examples omitted [trial]. The follow-up also reproduced a live-test failure without bus environment variables.

After: each reproduction passed after its fix, and the regressions are retained. The live 300 MiB Trial cap test also passed: only the Candidate died, its three attempts were Void, the Baseline finished, and the model Grader retained OOM score 1000.

cargo test --test experiments review_code_graders_raise_their_own_oom_score -- --exact --nocapture
cargo test --test experiments review_an_unreachable_manager_cannot_dispatch_uncapped_trials -- --exact --nocapture
XDG_RUNTIME_DIR=/run/user/1002 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus cargo test --test experiments review_a_cap_killed_tool_cannot_leave_a_successful_trial -- --exact --nocapture
cargo test --test user_config trial_memory_caps_reject_multiple_suffixes_before_work -- --exact
env -u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS cargo test --bin thirdshift scope::tests::a_live_scope_resolves_its_memory_events -- --exact --nocapture

Final validation passed: scripts/affected-tests issue-977 selected all 44 site tests and the full Rust suite: 2,518 passed, 58 skipped. The real user-bus environment enabled the live scope tests. /usr/bin/time -p cargo nextest run measured 2,039.87 seconds, including compilation (Nextest execution: 1,934.676 seconds) on this shared host. CI is fully green on e698c21d, including formatting, Clippy and the Linux Rust suite. Focused cargo check, Experiment and User config tests also passed; documentation defaults were checked in both source and generated HTML.

Test seams: the User config parser and CLI rejection before work; the Experiment CLI's retained Trial outcomes and Grader Scores; the scope launcher with real systemd caps; the internal --oom-exec command's inherited process score.

Both independent reviewers read all 23 changed files, including generated HTML directly. Changed files left unread: none on either axis. Generated-output exclusions: none. Reports, including each axis's final files-read list, are retained with the Session trace in .thirdshift-review-IS1GIg/standards.md and .thirdshift-review-IS1GIg/spec.md.

Unaddressed findings

Standards

None. All five Standards findings are addressed.

Spec

None. All three product findings and the follow-up validation finding are addressed.

Security

None. Security review of the issue-1015 diff (new src/scope.rs plus the Trial and Grader launch paths) against the resource-exhaustion and local-IPC guidance found no introduced finding with a failing proof-of-concept, so no Security fix was made here. Cap values reach systemd-run as separate argv entries under an allowlist validator enforced at config parse and env override; scope dispatch refuses to run uncapped when the user manager is unreachable; a memory_cap Void requires manager-owned kernel/journal evidence, never the agent's word.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Revert the code and remove the new [trial] settings before using an older binary that does not understand them.

Blast Radius: Experiments

Experiments require a reachable systemd user manager; enable lingering for logged-out work. Cap-killed Graders stop the Experiment for a cap adjustment and retry. Defaults use #1009's resident-size snapshot; a Trial's footprint beside two building Runs remains unmeasured.

Built with codex · gpt-6.1-sol · xhigh

Each Trial and each model Grader runs in a scope of its own, from
systemd-run --user --scope, with MemoryHigh, MemoryMax and MemorySwapMax
from the new [trial] User config section (defaults 1G, 2G, 1G from the
#1009 Harness measurements). The scope raises its own oom_score_adj to the
maximum; a Trial killed by its cap dies alone and is void with memory_cap,
from the scope's memory.events or journal evidence. Without a reachable user
manager Trials run directly, still raised, with a warning naming the
loginctl enable-linger step.
Route unscopped Trial and Grader fallbacks through the raising wrapper,
so a Grader without a user manager still raises its oom_score_adj.
Callers supply the machine caps instead of the launchers loading them,
and one Attempt bundle replaces the eight-argument trial command.
Merge the two launch types, name the shared oom-kill cause once, and
share the scope-availability check across the test suite.
…id=0e04ac92-0789-4332-9896-26905b35c395]: Subscription quota exhausted. Your usage window resets at 2026-10-24T03:28:16Z. (rate_limit_error))

2026-10-24T02:31:03Z, host stephens. Uncommitted work at the time of failure is included in this commit.
Preserve Trial and Grader memory scopes alongside the Trial network sandbox and allowlisting proxy. Resolve code-navigation and Experiment documentation to describe both changes.

Validation: cargo fmt --check; cargo clippy --all-targets -- -D warnings; scripts/affected-tests origin/issue-977 with the systemd user bus. All 44 site tests and 2,541 Rust tests passed; 58 existing Rust skips. Affected checks took 1,756.79 seconds including compilation (Nextest execution: 1,641.170 seconds).
@JacobStephens2
JacobStephens2 merged commit 42a7513 into issue-977 Oct 24, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1015 branch October 24, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant