Skip to content

Price Trial sessions by role and report paired cost and time - #1932

Merged
JacobStephens2 merged 11 commits into
issue-1484from
issue-1878
Nov 10, 2026
Merged

JacobStephens2 merged 11 commits into
issue-1484from
issue-1878

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Closes #1878

Trial sessions, retries, Resumes and fallbacks
  → starting Step + observed Model usage
  → Experiment's frozen, dated token prices
  → Session / Trial / role / Arm cost and time
  → paired median Candidate-to-Baseline ratios

Missing usage or an unpriced Model makes the Trial Void with a named accounting gap. Pre-run estimates use each exact Arm setup's history. Usage-limit pauses add no Trial time. Interrupted sessions retain their starting Step and elapsed time without publishing a completed span before launch. Priced Experiments checkpoint Session accounting before propagating a stop; the interruption regression also holds a worker in repository preparation so it exercises both started and unstarted attempts.

Evidence

Execution evidence

  • Before: cargo test --test experiments accounting:: -- --test-threads=2 — failed with unknown Experiment flag: --prices and a missing-usage Trial reported delivered.
    After: same command — 9 passed; 0 failed.
  • Before: cargo test --bin thirdshift session::trace::tests::trial_role_does_not_publish_an_ended_session_before_harness_launch -- --exact — failed because the initial Session already had an end time.
    After: same command — 1 passed; 0 failed.
  • The specific review reproductions also failed before their fixes: role process versus implement, legacy paired time null versus 2.0, and killed-session role time 0. Their original tests are retained and pass.
  • Before: cargo test --test experiments stop_and_resume_preserve_finished_trials_and_refuse_a_changed_bar -- --exact --nocapture — a controlled interruption retained 7 Session traces but counted only 6 sessions.
    After: same command passes; the accounting cases plus this regression pass under nextest (10 passed). The retention assertion now checks started attempts, while explicitly exercising an interrupted repository preparation.
  • A stop during code grading also retained 4 Session traces but initially counted only 2. Its focused regression now passes and retains the priced Session facts before propagating the stop.
  • Before: cargo test --test experiments resuming_a_cut_short_retry_keeps_the_bars_void_retry_budget -- --exact --nocapture — failed with 6 session calls versus 8 because the new stop checkpoint consumed a completed-attempt retry.
    After: incomplete accounting checkpoints preserve the original eight-call retry assertion. This regression and both session/grading stop regressions pass together under nextest (3 passed).
  • Final local scripts/affected-tests origin/issue-1484 on e3e69fb0 passed 67 site tests and 169 Console tests. Its Rust run stopped at the unchanged Console idle-cache test (unchanged request reread 107775 bytes); 393 tests had passed. cargo test --test console idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible -- --exact --nocapture reproduced that local failure alone on the preceding head. The test and cache implementation are unchanged by this PR.
  • Full CI for 5680f90a passed: 3,964 Rust tests, 91 configured skips, plus formatting, all-target clippy, tooling, site and Console checks. Rust test execution took 1151.5 seconds. The later Custom process reconciliation passes all 37 focused accounting/process/Trial/stop checks. Formatting and all-target clippy also pass on final head e3e69fb0; its full CI is still running.

Test seams: Experiment CLI JSON and retained reads; the terminal pre-run question; Session trace lifecycle.

Visual evidence

The same expanded Experiment documentation and scroll positions before/after, at both viewports.

Architecture — 1440×900

Before

After

Architecture — 390×844

Before

After

Manual — 1440×900

Before

After

Manual — 390×844

Before

After

Review coverage

Reviewed against issue-1484 pinned at dfcdeca5d9ca5251be634d0f345e599715e2ac53: Standards raised 3 findings; Spec raised 3. All six were addressed. Every behavioral finding was reproduced with its written command before deciding it.

In this PR's 29 changed files, Standards left tests/experiments/code_graders.rs unread. Spec left that file and src/session/trace/tests.rs unread: these regressions were added after the axes' single coverage follow-ups. No generated-output exclusions.

The base subsequently advanced through #1927 and #1928; merges 5680f90a and e3e69fb0 retain process Graders, Custom process worktrees and starting-Step role accounting. All 37 focused pricing/process/Trial/stop regressions pass after reconciliation. The original pinned comparison now has 58 changed paths. Both axes left these inherited base files unread:

  • docs/eval-tasks.md
  • site/tests/pages.test.cjs
  • src/evals.rs
  • src/evals/arms.rs
  • src/evals/arms/frozen.rs
  • src/evals/bug_origin.rs
  • src/evals/experiment/grading.rs
  • src/evals/experiment/grading/process.rs
  • src/evals/experiment/retention.rs
  • src/evals/format.rs
  • src/evals/import.rs
  • src/evals/process_tasks.rs
  • src/evals/process_tasks/proposals.rs
  • src/evals/repository.rs
  • src/processes/review.rs
  • src/sandbox.rs
  • src/sandbox/code_grader.rs
  • src/sandbox/triage.rs
  • src/skills.rs
  • tests/eval_arms.rs
  • tests/evals/process_tasks.rs
  • tests/evals/regressions/executable_overlay.py
  • tests/experiments/result_format.rs
  • tests/experiments/sandbox.rs
  • tests/trial/process.rs

Standards also left inherited docs/trial.md unread; Spec read that file. These inherited changes were not sent through another coverage follow-up. Reports retain each axis's complete final files-read list and all remaining gaps.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Blast Radius: Experiments

New Experiments require complete usage and exact observed-Model prices; missing accounting becomes Void. Older Experiments keep their recorded pricing policy and outcomes.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh

…ption

The supervisor skipped its repository integrity walk whenever the Arm protocol's void cause was interrupted, including when the outcome was still success. A Trial could edit a reviewed input, report success with that cause, and keep the mutated result. The walk is skipped only when the attempt is already void for interruption.
Preserve Trial role pricing and accounting-gap Voids alongside process Trial leak detection and fix-leaked Voids. Retain both features in the result contract and domain-to-code index.
@JacobStephens2
JacobStephens2 merged commit 2b81a53 into issue-1484 Nov 10, 2026
16 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1878 branch November 10, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant