[JacobStephens2]

Summary

Experiments can compare complete Built-in or Custom Ticket processes on past merged work, including implementation and Security review, and grade the Trial's own final commit.

merged Ticket → screen reference tests at base and merge
             → freeze each Arm's build, Process and home skills
             → run Steps locally, stopping before publication
             → overlay reference tests on the Trial's final commit
             → grade offline + check transcripts + price Sessions
             → paired quality, cost and time Verdict

The shared Ticket-source seam preserves factory execution and supplies local Trial inputs without GitHub, a Claim or a pull request. Trials retain Steps, Sessions, traces, delivered/stopped-to-ask/failed outcomes and usage under their Experiment; they do not enter factory run lists or metrics. Stopped-to-ask fails quality; outside failures and incomplete Session accounting remain Void.

Process task proposals retain real merged-work provenance, the host environment, separate reference tests, changed shared fakes and fail-to-pass/pass-to-pass selectors. Preparation screens the tests at both commits and refuses changed inline Rust tests. The suite selects three Trials and declares its Factory skill, Process, prompt, Harness, Model and Effort dependencies. Building and measuring the real task inventory remains the separate post-release work described by the Spec.

Arms freeze Custom processes and referenced home skills by content hash and check them before every Trial. Review fixes also include those hashes in Experiment identity and cost history, so editing the same source path starts a new comparison. Unchanged inputs resume their frozen Experiment; restore an older Custom input's bytes to resume its older comparison. Its saved results remain readable by ID.

The code Grader applies reference tests to the final committed work and shares one Bar deadline across snapshot preparation and all selectors. Session accounting uses a dated frozen price table, observed Models and Step roles. Paired median cost/time ratios, optional fixed ceilings and cheaper/faster Verdict text accompany quality. Transcript leak hits await auditable human decisions; confirmed leaks void the Trial. CLI and Console expose the same retained facts, including format-8 Experiment JSON and its corrected OpenAPI schema.

Closes #1484

Seams under test

• Experiment CLI JSON through the existing fixture: changed Custom inputs select a new comparison; combined selector work respects the fixed grading timeout.
• Console Experiment JSON/OpenAPI and detail HTML: published formats and Bar wording agree with served results.
• Console HTTP process-I/O checks retain their original assertion with host Cron jobs isolated at the command seam.
• Existing Spec acceptance seams: local --trial, evals prepare, evals propose-process, Arm preparation, Experiment resume, and Console Trial/trace/leak-decision views.

Evidence

Execution evidence

• cargo test --test experiments process::edited_custom_process_starts_a_new_experiment -- --exact --nocapture
  Before: assertion left != right failed: changed Process bytes must select a new frozen Arm; both IDs were identical.
  After: 1 passed; 0 failed.
• cargo test --test experiments process::process_grading_shares_one_bar_timeout_across_selectors -- --exact --nocapture
  Before: combined grading must stop at the Bar timeout; the Score was passed: true after two four-second selectors under a six-second Bar.
  After: 1 passed; 0 failed; a timed-out check fails the Score; later selectors stay unstarted after the shared budget expires. The existing selector-timeout regression now asserts that same behavior.
• cargo test --test console experiments::openapi_experiment_format_matches_served_list_and_detail -- --exact --nocapture
  Before: /api/v1/experiments advertised Number(6) while the served format was Number(7).
  After: 1 passed; 0 failed for both list and detail schemas. The added Arm digest advances both the view and its schema to format 8 under the documented versioning rule.
• cargo test --test console experiments::console_experiment_does_not_claim_efficiency_is_outside_the_bar -- --exact --nocapture
  Before: the rendered comparison contained Efficiency stays outside the Bar with a recorded cost ceiling.
  After: 1 passed; 0 failed; the text explains the ceilings and distinguishes frozen-price accounting from Harness-reported usage.
• Removed the duplicate hard-kill role-only regression; the stronger retained test covers that same role plus positive Session and role time.
• cargo test --test console idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible -- --exact --nocapture
  Before: unchanged request reread 107775 bytes. The record was read only once; host crontab -l and PAM startup caused the extra I/O.
  After: 1 passed; 0 failed after giving the shared Console fixture an empty-crontab fake. The original 48 KiB assertion and cache behavior are unchanged.
• Validation: scripts/affected-tests main passed Go, TypeScript, site, production-build and Console-browser checks. Its Rust phase exposed the old selector-timeout assertion described above; after updating that assertion, cargo nextest run --no-fail-fast passed all 3,988 tests (91 skipped) in 2,113.64s wall time, including compilation with warm dependencies and four workers. All review regressions pass. Python tooling: python3 -m unittest discover -s scripts -p 'test_*.py' — 41 passed with Python 3.11. cargo fmt --check and git diff --check passed.

Visual evidence

Matched Before (main) and After Console Experiment pages were captured and inspected at 1440×900 and 390×844, with the Arm comparison and Bar open. Four PNGs are retained in .thirdshift-review-rq0eHq/visual/ beside the Session's review artifacts. The captures show the corrected comparison text and fixed resource ceilings. Media hosting via a git-pushed Gist was omitted under the no-push instruction.

Review coverage

The independent reviews cover the entire main...HEAD comparison, including the merged Fleet and prototype ancestry. Reports and final files-read lists are retained beside the Session trace in .thirdshift-review-rq0eHq/standards.md and spec.md; counts in summary.json are Standards 2 and Spec 4, including fixed findings. OpenAPI was raised on both axes.

• Standards read 277 of 278 changed files. prototype/go/factory-skills.json was intentionally not read: its sources are the unchanged skills/**/* tree, with generator prototype/go/bundle-skills.py and consumer prototype/go/skills.go reviewed. python3 prototype/go/bundle-skills.py --check exited 0; go test ./... passed, including the inspected TestTicketRunLinksEmbeddedSkillsAndStreamsSession consumer assertions. This is the sole generated-output exclusion.
• Spec read all 278 changed files, with no exclusions. The generated manual was inspected directly on both axes. There are no other unread changed files.

Unaddressed findings

Standards

None. Both findings addressed.

Spec

None. All four findings addressed.

Security

None. The process Grader's script-directory standard-library bypass was fixed. A python3 tests/check.py selector put the script directory ahead of the standard library, so a Trial could commit tests/pathlib.py and pass a broken commit. The import guard now moves every path inside the repository behind the standard library. Proof: cargo test --bin thirdshift sandbox::code_grader::tests::a_script_directory_cannot_shadow_the_standard_library -- --exact failed before the fix (script directory shadowed pathlib: exit status: 0) and passed after it. The same command on the pinned merge base ran 0 tests and exited 0; that grading path is not on the base.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Code rollback is available; saved Experiment records remain retained. Experiment JSON clients must recognize format 8 for the added Arm digest; private metadata stays at format 1. Changed Custom process content now receives a distinct Experiment identity.

Blast Radius: Factory

Shared Ticket Process execution, Experiment preparation/grading/accounting, retained result contracts and Console views. The remaining real-task rollout needs the released build.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh

Tickets

• Put the Process engine's GitHub, Claim and pull-request use behind one seam #1873 landed with Isolate the Process engine’s Ticket source and publication #1907
• Run the Built-in Ticket run as a process Trial, up to Delivery #1874 landed with Run Built-in Ticket processes as Experiment Trials #1919
• Grade a process Trial's commit with the fix's tests overlaid #1875 landed with Grade process Trials with the reference fix's tests #1927
• Build and screen process Eval tasks from merged pull requests #1876 landed with Build and screen process Eval tasks from merged Tickets #1926
• Let an Arm name a Custom process and home skills, frozen by hash #1877 landed with Freeze Custom process Arms and home skills for Experiment Trials #1928
• Price each Trial's sessions per role and report paired cost and time #1878 landed with Price Trial sessions by role and report paired cost and time #1932
• Add a cost and time ceiling to the Bar, and a cheaper/faster Verdict line #1879 landed with Add cost and time ceilings and cheaper/faster Experiment Verdicts #1938
• Check Trial transcripts for the real fix #1880 landed with Check process Trial Session evidence for the real fix #1934