Repository navigation
Price Trial sessions by role and report paired cost and time - #1932
Merged
Merged
Conversation
…ption The supervisor skipped its repository integrity walk whenever the Arm protocol's void cause was interrupted, including when the outcome was still success. A Trial could edit a reviewed input, report success with that cause, and keep the mutated result. The walk is skipped only when the attempt is already void for interruption.
Preserve Trial role pricing and accounting-gap Voids alongside process Trial leak detection and fix-leaked Voids. Retain both features in the result contract and domain-to-code index.
This was referenced Nov 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1878
Missing usage or an unpriced Model makes the Trial Void with a named accounting gap. Pre-run estimates use each exact Arm setup's history. Usage-limit pauses add no Trial time. Interrupted sessions retain their starting Step and elapsed time without publishing a completed span before launch. Priced Experiments checkpoint Session accounting before propagating a stop; the interruption regression also holds a worker in repository preparation so it exercises both started and unstarted attempts.
Evidence
Execution evidence
cargo test --test experiments accounting:: -- --test-threads=2— failed withunknown Experiment flag: --pricesand a missing-usage Trial reporteddelivered.After: same command —
9 passed; 0 failed.cargo test --bin thirdshift session::trace::tests::trial_role_does_not_publish_an_ended_session_before_harness_launch -- --exact— failed because the initial Session already had an end time.After: same command —
1 passed; 0 failed.processversusimplement, legacy paired timenullversus2.0, and killed-session role time0. Their original tests are retained and pass.cargo test --test experiments stop_and_resume_preserve_finished_trials_and_refuse_a_changed_bar -- --exact --nocapture— a controlled interruption retained 7 Session traces but counted only 6 sessions.After: same command passes; the accounting cases plus this regression pass under nextest (
10 passed). The retention assertion now checks started attempts, while explicitly exercising an interrupted repository preparation.cargo test --test experiments resuming_a_cut_short_retry_keeps_the_bars_void_retry_budget -- --exact --nocapture— failed with 6 session calls versus 8 because the new stop checkpoint consumed a completed-attempt retry.After: incomplete accounting checkpoints preserve the original eight-call retry assertion. This regression and both session/grading stop regressions pass together under nextest (
3 passed).scripts/affected-tests origin/issue-1484one3e69fb0passed 67 site tests and 169 Console tests. Its Rust run stopped at the unchanged Console idle-cache test (unchanged request reread 107775 bytes); 393 tests had passed.cargo test --test console idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible -- --exact --nocapturereproduced that local failure alone on the preceding head. The test and cache implementation are unchanged by this PR.5680f90apassed: 3,964 Rust tests, 91 configured skips, plus formatting, all-target clippy, tooling, site and Console checks. Rust test execution took 1151.5 seconds. The later Custom process reconciliation passes all 37 focused accounting/process/Trial/stop checks. Formatting and all-target clippy also pass on final heade3e69fb0; its full CI is still running.Test seams: Experiment CLI JSON and retained reads; the terminal pre-run question; Session trace lifecycle.
Visual evidence
The same expanded Experiment documentation and scroll positions before/after, at both viewports.
Architecture — 1440×900
Architecture — 390×844
Manual — 1440×900
Manual — 390×844
Review coverage
Reviewed against
issue-1484pinned atdfcdeca5d9ca5251be634d0f345e599715e2ac53: Standards raised 3 findings; Spec raised 3. All six were addressed. Every behavioral finding was reproduced with its written command before deciding it.In this PR's 29 changed files, Standards left
tests/experiments/code_graders.rsunread. Spec left that file andsrc/session/trace/tests.rsunread: these regressions were added after the axes' single coverage follow-ups. No generated-output exclusions.The base subsequently advanced through #1927 and #1928; merges
5680f90aande3e69fb0retain process Graders, Custom process worktrees and starting-Step role accounting. All 37 focused pricing/process/Trial/stop regressions pass after reconciliation. The original pinned comparison now has 58 changed paths. Both axes left these inherited base files unread:docs/eval-tasks.mdsite/tests/pages.test.cjssrc/evals.rssrc/evals/arms.rssrc/evals/arms/frozen.rssrc/evals/bug_origin.rssrc/evals/experiment/grading.rssrc/evals/experiment/grading/process.rssrc/evals/experiment/retention.rssrc/evals/format.rssrc/evals/import.rssrc/evals/process_tasks.rssrc/evals/process_tasks/proposals.rssrc/evals/repository.rssrc/processes/review.rssrc/sandbox.rssrc/sandbox/code_grader.rssrc/sandbox/triage.rssrc/skills.rstests/eval_arms.rstests/evals/process_tasks.rstests/evals/regressions/executable_overlay.pytests/experiments/result_format.rstests/experiments/sandbox.rstests/trial/process.rsStandards also left inherited
docs/trial.mdunread; Spec read that file. These inherited changes were not sent through another coverage follow-up. Reports retain each axis's complete final files-read list and all remaining gaps.Unaddressed findings
Standards
None.
Spec
None.
Security
None.
Security review outcome
No unaddressed introduced findings.
Merge Danger
Door: two-way
Blast Radius: Experiments
New Experiments require complete usage and exact observed-Model prices; missing accounting becomes Void. Older Experiments keep their recorded pricing policy and outcomes.
Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh