[JacobStephens2]

Summary

Process Eval tasks now run the Built-in Ticket run through the Experiment, retaining every Step and session without publishing factory work.

real/host process task + Arm's Built-in process, Harness, Model and Effort
  base checkout → implement → security-review → stop before Delivery
    Experiment owns Steps, sessions, Trace, usage and times
  delivered → pass
  stopped-to-ask / failed → fail
  external failure → existing Void rules

Process grading is outcome-only for this Ticket. Existing Code review and Spec review grading remains intact. Reference fixes stay outside implementing Trials; repository integrity applies to review Trials.

Closes #1874

Evidence

Execution evidence

• Before: cargo test --test experiments process::built_in_ticket_trials_deliver_and_keep_all_steps_and_sessions -- --exact failed with left: String("failed"), right: "delivered" when repository preparation exposed the reference fix.
  After: the same test passes, including committed edits, both Arms' Steps, four sessions, Step ancestry in the exported Trace, outcome counts and factory-list isolation.

• Before: python3 .thirdshift-review-u6pc7l/run-spec-regression.py process::spec_process_outcome_counts_exclude_failed_review_trials failed: failed review Trial contaminated Process Trial outcomes, left: Number(1), right: 0.
  After: the retained regression passes; process counts exclude failed review Trials.

• Before: python3 .thirdshift-review-u6pc7l/run-spec-regression.py process::spec_a_markdown_final_question_is_stopped_to_ask failed: left: String("delivered"), right: "stopped_to_ask".
  After: the retained regression passes and the Score is failed.

• Before: PLAYWRIGHT_BROWSERS_PATH=/tmp/thirdshift-1874-browser-compatible node --test --test-name-pattern='process Trial Peeks' console/tests/experiments.test.cjs timed out looking for the delivered Trial's Baseline Trial 1: Passed button.
  After: the focused browser regression passes, showing process Steps, outcomes and delivered Scores.

• Before: cargo test --bin thirdshift evals::experiment::execution::tests::a_delivered_review_that_changes_the_repository_is_voided -- --exact failed with a delivered review Trial kept a mutated repository.
  After: the same test voids that review Trial as repository_changed and scores it as a fail. Process Trials that change the repository still deliver.

Test seams: the Experiment CLI and retained evidence/export commands; the direct Trial protocol; the Console's Experiment page and Trial Peek.

Validation: cargo clippy --all-targets -- -D warnings passes. scripts/affected-tests issue-1484 passes the 67 site tests and locked build, then stops at the Console suite: 165 passed, 4 failed in unchanged Fleet/Inbox flows (sign-in connection refusal and an Undo timeout with two resulting count assertions). All four pass when rerun directly: PLAYWRIGHT_BROWSERS_PATH=/tmp/thirdshift-1874-browser-compatible node --test --test-concurrency=1 --test-name-pattern='Fleet starts, Passes and config|inline Dismiss leaves|every section folder fits|tab title keeps' console/tests/fleet-control.test.cjs console/tests/inbox.test.cjs (4 passed, 19.77s). The new process UI test passes. Browser validation uses installed Chromium 1247 through private compatibility paths; browser files are serialized and Rust uses three test workers.

/usr/bin/time -p env RUST_TEST_THREADS=3 cargo test ran for 485.07s including compilation: unit tests and the first integration targets passed, then an unchanged Bug origins test required tomllib, absent from the host's Python 3.9. Retrying that target and subsequent integration targets with installed Python 3.11 passed Bug origins and reached Console: 263 passed, 1 failed, 1 ignored. The remaining failure is cargo test --test console idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible -- --exact --nocapture: unchanged request reread 107775 bytes. Running that exact test binary against a fresh build of PR base 2b1dbd0e produces the identical 107775-byte failure (exit 101), confirming it predates this PR. The temporary baseline checkout was removed and the published binary restored. Local broad Rust validation is therefore incomplete; CI runs the complete suite on eb823688. Its site and Console jobs have passed; its Rust test job is still running at this update.

Visual evidence

Experiment Trial Peek, desktop 1440×900, same synthetic fixture and selected Trial:

Experiment Trial Peek, mobile 390×844, same fixture and selected Trial:

Review

Fixed point: issue-1484 (9283d0b8), including its inherited #1873 prefactor. Standards raised 1 heuristic finding; Spec raised 2 behaviour findings. All three were accepted and fixed; both behaviour reproducers failed as predicted before their fixes. Security review found 1 introduced finding and fixed it; the regression is a_delivered_review_that_changes_the_repository_is_voided. Full reports and final files-read lists are retained in .thirdshift-review-u6pc7l/standards.md and spec.md, with summary.json recording {"standards":1,"spec":2}.

Both axes read all 41 changed paths, including generated site/docs/index.html; no generated output was excluded. Their final independent pass covered c6f9e943. Later tooling-only hunks in src/evals/experiment/store.rs (text formatting) and src/processes/source.rs (exhaustive Step matching) were checked by the implementing agent and validation, and were not reread by either independent reviewer. These are the two paths with final hunks outside the independent review.

Unaddressed findings

• Standards: None. S1's duplicated setup now uses prepare_local_process.
• Spec: None. SP1 and SP2 have retained regression tests and fixes.
• Security: None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Existing task manifests and review Experiment identities remain compatible. New process outcomes require a build that understands process Trials.

Blast Radius: Experiments

Also exercises the shared local Ticket source; existing factory Process and Ticket-run tests cover its production adapter.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh