Skip to content

Grade process Trials with the reference fix's tests - #1927

Merged
JacobStephens2 merged 4 commits into
issue-1484from
issue-1875
Nov 10, 2026
Merged

JacobStephens2 merged 4 commits into
issue-1484from
issue-1875

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

process Trial final commit
  export committed files under the Bar timeout
  overlay reference fix's tests and changed shared fakes
  run fail-to-pass selectors offline in bwrap
  run pass-to-pass selectors offline in bwrap
  pass only when the Trial and both groups pass

Review Trials keep grading their sealed pre-session snapshot. Process tasks declare a required process_grading table with files and argument-vector selectors.

Closes #1875

Test seams

  • thirdshift experiment: manifest admission, process Trials, and retained Score checks, commands and reasons.
  • The same command with pass-through Trial launchers and real bwrap Graders: network isolation and the Bar timeout.

Evidence

Execution evidence

  • Before: cargo test --test experiments process::a_delivered_trial_without_the_fix_fails_its_selector -- --exact --nocapture failed with left: Bool(true), right: false: the delivered broken fixture incorrectly passed.
    After: the same command passed: 1 passed; 0 failed (11.33s).
  • Before: timeout --kill-after=5s 40s cargo test --test experiments process::process_grader_export_obeys_the_bar_timeout -- --exact --nocapture reached running 1 test and hung until the guard exited 124.
    After: the same command passed: 1 passed; 0 failed (13.23s, exit 0), returning a failed timeout check. The review regression is retained.
  • cargo test --test experiments process:: -- --test-threads=3 --nocapture passed all 14 checks on the final commit (52.32s, 0 skipped), including real bwrap isolation and selector timeout, export timeout, neighboring regressions, test/fake tampering, and exclusion of uncommitted repairs.
  • cargo check --bin thirdshift passed. npm test --prefix site passed all 67 checks with 0 skipped.
  • cargo fmt --check and cargo clippy --bin thirdshift -- -D warnings passed on the final commit.
  • /usr/bin/time -p scripts/affected-tests origin/issue-1484 selected the site checks and full Rust suite, with CARGO_BUILD_JOBS=2 and RUST_TEST_THREADS=3. Site checks passed (99.29s), as did 1,663 unit tests and the completed integration groups. The run stopped in an unchanged Bug-origin regression because the shell's Python 3.9 lacks tomllib. Total wall time: 1008.31s; Rust command: 908.57s, including 5m25s compilation with warm dependency caches.
  • With an isolated PATH selecting installed Python 3.11.13, cargo test --test bug_origins settlement::historical_review_and_reclosed_origin_regressions -- --exact --nocapture passed (1 passed, 2.68s). No source change was needed.
  • Initial CI stopped in the existing Fleet Stop test (died before recording an outcome versus interrupted). The exact case passed alone on this commit (1 passed, 14.25s). The full CI retry passed for head b1bba8bf: 3,946 passed, 91 skipped, across 95 binaries; test execution 981.251s, test compilation 59.22s. Formatting, all-target clippy, Python tooling, site and Console checks also passed. This successful exact-head CI supplies the full Rust result, without repeating the full suite locally.

Visual evidence

Updated documentation: paired desktop and mobile captures

Homepage, Experiments expanded — 1440×900

Before

After

Homepage, Experiments expanded — 390×844

Before

After

How it works, process Trials — 1440×900

Before

After

How it works, process Trials — 390×844

Before

After

Architecture, Trial protocol — 1440×900

Before

After

Architecture, Trial protocol — 390×844

Before

After

User manual, Experiments expanded — 1440×900

Before

After

User manual, Experiments expanded — 390×844

Before

After

Merge Danger

Door: two-way

Blast Radius: Experiments

Existing process manifests must add process_grading; reference overlay failures become failed checks. No data migration is performed.

Unaddressed findings

Standards

None. The independent reviewer raised 0 findings across initial review and follow-up.

Spec

None skipped. The independent reviewer raised 1 finding, unbounded final-commit export. Its exact reproduction failed before the fix and passed after commit b1bba8bf; the reviewer confirmed resolution.

Security

None. One introduced finding, a committed module shadowing the Python standard library and forging a passing Score, is fixed in this review. Its regression test failed before the fix and passed after commit fc665173.

Review coverage

Both reviewers inspected every changed file, including the generated manual hunks, against the fetched issue-1484 fixed point 8ad2733cfb7f8d3b9869a2c1b8f00b2eb224b447. No changed files were unread and no generated output was excluded. Reports and the pre-disposition counts ({"standards":0,"spec":1}) are retained with the Session trace.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh

Security review outcome

No unaddressed introduced findings.

A Trial could commit a module such as pathlib.py and make the overlaid
reference tests import it, so a broken commit still passed. Process
grading now searches the repository after the standard library. Review
Graders are unchanged.
Preserve the process Grader documentation alongside process task proposals. Have proposals and preparation share the required process_grading table used by the Grader, retaining the proposal report format.

Validation: Eval and process Trial tests, site and Console browser tests, cargo check, formatting, and all-target clippy pass. Full Rust validation stops at the Console idle IO regression already documented in #1926 (107775 bytes); its isolated rerun reproduces the failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant