Skip to content

Propose Bug origins for closed bugs during Scoring - #1276

Merged
JacobStephens2 merged 424 commits into
issue-978from
issue-1028
Oct 24, 2026
Merged

JacobStephens2 merged 424 commits into
issue-978from
issue-1028

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

Scoring run
  closed bugs (first proposal run: all history; later runs: window)
  sandbox: fix + earlier history → introducing commit or none, with evidence
  GitHub commit → merged pull requests → local Run / Built-with line
  save proposed origins with the window; no metrics effect before #1029

Closes #1028

Includes dependency merge d64ec50 from main for #977's Eval sandbox and #1008's plan-limit detection. Bug-origin sessions and model Graders now share private-session setup, caps, recording and native quota observation.

Evidence

  • Dependency merge (d8c405e): reconciled Your Scores and calibration #1027's human Scores and calibration with Bug-origin proposals, retaining both checkpoint fields and sandbox launch interfaces. cargo check, cargo fmt --check and cargo clippy --all-targets -- -D warnings pass. RUST_TEST_THREADS=8 /usr/bin/time -p scripts/affected-tests origin/issue-978 passes: 2,859 Rust tests, 62 ignored fixture/platform tests, 50 browser checks and 3 release-generator tests; 1267.14 s wall time, including compilation. CI is green for exact head d8c405e.

  • Before: named_history_commit_maps_through_github_to_the_other_hosts_run failed because Scoring retained no proposal.
    After: cargo test --test bug_origins passes 10 tests, including real Git history, local/remote Run attribution, no-commit evidence, Security-fix exclusion, pagination, 1,001-bug backfill, legacy checkpoints and live limit stops. One subprocess-only fixture entry point is ignored.

  • Before: the reviewer's exact cutoff test returned 0 bugs instead of 1; the exact native Codex test incorrectly completed Scoring and advanced its window.
    After: both retained regressions pass. The native test ran with real bwrap, without skipping (1 passed, 0 ignored, 1.86 s):

    cargo test --bin thirdshift github::execution_tests::first_bug_backfill_keeps_closures_in_the_cutoffs_second -- --exact
    cargo test --test bug_origins sandboxed_bug_origin_stops_at_codex_native_plan_limit -- --exact --nocapture
  • Existing Scoring tests: 19 passed, one fixture entry point ignored. Existing model-Grader tests: 12 passed. Prompt tests: 12 passed. The paused fake GitHub search now releases its state lock, preserving the live Stop/locking test after the dependency merge.

  • Final: CI run 40133979982 passed for exact head 17f1158, including the full Rust suite, formatting, clippy, Python tooling, Console and site checks. All PR checks are green. Local formatting, clippy and all 16 Python tooling tests also passed.

  • Local selector evidence: /usr/bin/time -p scripts/affected-tests issue-978 took 764.51 s, including compilation and 50 passing browser checks, then caught the existing help-size regression (66 lines versus 65). 17f1158 compacts the layout; its exact existing test passes. RUST_TEST_THREADS=8 /usr/bin/time -p scripts/affected-tests issue-978 ran 942.52 s, including compilation and 50 passing browser checks, and remained green through Security-review integration. It was stopped only after the exact-head full CI suite passed, reusing that result under docs/testing.md rather than duplicating the remaining checks.

  • Pickup verification: model mapping is in sync at source 3cd68bb; normal check passes (4 tests, 1 explicit literal assertion ignored). The literal property and both safeguard-removal checks exit 101 as expected. The documented pre-Claim duplicate-dispatch counterexample remains; the green narrower check does not prove the literal property. The skill records this without changing the model or property.

Test seams

  • thirdshift score / thirdshift scores --json, real Git and fake GitHub/Harness executables; shared sandbox and systemd launchers at their existing executable seams.
  • Retained Run lookup through Scoring and an actual factory Run's Command record.
  • GitHub closed-bug search with an isolated executable and fixed timestamp bounds.

Unaddressed findings

Standards

None. The duplicated private-session lifecycle was consolidated in src/session/private.rs.

Spec

None. Both demonstrated behavioral defects were fixed and their exact regressions retained.

Security

None. A symlink or other non-regular session file is rejected before Scoring or the model Grader stores it (95d66bb).

Security review outcome

No unaddressed introduced findings.

Review coverage

Independent Standards and Spec reviews used issue-978 (8905eb4) as the fixed point, with one coverage follow-up each. Reports, including final files-read lists, are retained beside the Session at .thirdshift-review-2q6q8J/standards.md and spec.md. Large imported Console/Experiment changes retain coverage gaps. The final verification-skill sync and compact-help layout fix were not sent for another review; both paths are disclosed below.

Every changed path with a review gap or generated exclusion is named below. Gap includes unread or partly inspected changed hunks; covered means the axis reports no remaining gap. Generated is intentionally excluded from full reading: Standards inspected sources/generator and meaningful rendering checks, with UPDATE_PROMPTS=1 cargo test --bin thirdshift prompts_page passing 12 tests; handwritten prompt-page content and the Bug-origin card were inspected. Spec claimed no generated exclusions. Sources: src/prompt.rs, src/prompts_page.rs, Harness arguments and embedded skills/; normal prompt and docs-generation checks also passed.

Changed files with review gaps or exclusions
File Standards Spec
.agents/skills/verify-pickup/SKILL.md gap gap
Cargo.lock gap gap
GLOSSARY.md gap covered
README.md gap gap
console/README.md gap gap
console/reproductions/span-tree-review.cjs gap gap
console/reproductions/waterfall-fractional.py gap gap
console/reproductions/waterfall-stage.py gap gap
console/tests/experiments.test.cjs gap gap
console/tests/fixtures/experiments.cjs gap gap
console/tests/inbox.test.cjs gap gap
console/tests/logs.test.cjs gap gap
console/tests/resources.test.cjs gap gap
console/tests/set-password.test.cjs gap gap
console/tests/spans.test.cjs gap gap
console/tests/time.test.cjs gap gap
console/tests/waterfall.test.cjs gap gap
docs/adr/0023-the-inbox-reads-github-to-learn-the-day-shift-acted.md covered gap
docs/eval-tasks.md gap gap
docs/grill-with-docs/1252-1253-bar-and-rechecks.md gap gap
docs/grill-with-docs/919-dismiss-and-github-state.md gap gap
docs/grill-with-docs/done/1251-spec-review-trial-inputs.md gap gap
docs/guide.md gap covered
docs/research/eval-sandboxes.md gap gap
docs/trial.md gap gap
prompts/architecture-review.md covered gap
prompts/ci-fix-repair-inherited.md covered gap
prompts/ci-fix-repair.md covered gap
prompts/code-review.md covered gap
prompts/conflict-repair.md generated gap
prompts/continuation-pr.md generated gap
prompts/continuation.md generated gap
prompts/foreign-conflict-repair.md generated gap
prompts/fresh.md generated gap
prompts/grader.md generated gap
prompts/resume.md covered gap
prompts/review-repair.md covered gap
prompts/security-audit.md covered gap
scripts/generate-release.py gap gap
scripts/release-experiment.py gap gap
scripts/release.sh gap gap
scripts/test_generate_release.py gap gap
site/architecture/index.html gap covered
site/docs/index.html gap covered
site/how-it-works/index.html gap covered
site/index.html gap covered
site/prompts/index.html generated covered
site/share-card.png covered gap
site/tests/docs.test.cjs gap gap
site/tests/pages.test.cjs gap gap
src/asks.rs gap gap
src/base_fix.rs gap covered
src/child_run.rs gap gap
src/child_run/execution_tests.rs gap covered
src/ci.rs gap gap
src/config.rs gap gap
src/console.rs gap gap
src/console/accounts/tests.rs covered gap
src/console/assets/console.css gap gap
src/console/assets/console.js gap gap
src/console/assets/experiments.css gap gap
src/console/assets/experiments.js gap gap
src/console/assets/look.js covered gap
src/console/assets/spans.css gap gap
src/console/assets/spans.js gap gap
src/console/assets/stages.js gap gap
src/console/auth/password.rs gap gap
src/console/changes.rs gap gap
src/console/dismiss.rs covered gap
src/console/experiments.html covered gap
src/console/experiments.rs gap gap
src/console/experiments/render.rs gap gap
src/console/github.rs covered gap
src/console/inbox.html gap covered
src/console/live.rs gap gap
src/console/logs.rs gap gap
src/console/peek.rs gap gap
src/console/resources.html gap covered
src/console/resources.rs covered gap
src/console/row.html gap covered
src/console/subscription.rs gap gap
src/console/subscription/tests.rs gap gap
src/console/time.html gap gap
src/console/watching.rs gap gap
src/console/waterfall.html gap gap
src/console/waterfall.rs gap gap
src/delivery.rs gap gap
src/evals.rs gap gap
src/evals/approval.rs gap gap
src/evals/arms.rs gap gap
src/evals/experiment.rs gap gap
src/evals/experiment/evidence.rs gap gap
src/evals/experiment/grading.rs gap gap
src/evals/experiment/integrity.rs gap gap
src/evals/experiment/limits.rs gap gap
src/evals/experiment/model_grader.rs covered gap
src/evals/experiment/options.rs gap gap
src/evals/experiment/rechecks.rs gap gap
src/evals/experiment/release.rs covered gap
src/evals/experiment/store.rs gap gap
src/evals/experiment/trace.rs gap gap
src/evals/experiment/usage.rs gap gap
src/evals/experiment/workers.rs gap gap
src/evals/format.rs gap gap
src/evals/import.rs gap gap
src/evals/repository.rs gap covered
src/evals/suite.rs gap gap
src/github/advisories.rs gap gap
src/github/advisories/description.rs gap gap
src/github/advisories_tests.rs gap gap
src/github/quota.rs gap gap
src/harness/codex/patches.rs gap gap
src/harness/interpretation_tests.rs gap gap
src/harness/muse.rs gap covered
src/launch.rs gap covered
src/logs/command_record.rs gap covered
src/logs/convert.rs gap gap
src/logs/convert/readings.rs gap gap
src/logs/dismiss.rs gap gap
src/logs/export.rs gap gap
src/logs/github.rs gap gap
src/logs/owner.rs gap gap
src/logs/reading/cache.rs gap gap
src/logs/reading/captured_activity.rs gap gap
src/logs/reading/catalog.rs gap gap
src/logs/reading/health.rs gap gap
src/logs/reading/history.rs gap gap
src/logs/reading/inbox.rs gap gap
src/logs/reading/inventory.rs gap gap
src/logs/reading/inventory_tests.rs gap gap
src/logs/reading/reader.rs gap gap
src/logs/reading/reader_tests.rs gap gap
src/logs/reading/tests.rs gap gap
src/logs/session.rs gap gap
src/logs/spans.rs gap gap
src/logs/spans/session.rs gap gap
src/logs/status.rs gap gap
src/logs/status/inbox.rs gap gap
src/logs/status/inspect.rs gap gap
src/logs/status/time.rs gap gap
src/logs/status/watching.rs gap gap
src/logs/status/waterfall.rs gap gap
src/logs/status/waterfall/matching.rs gap gap
src/logs/status/waterfall/trace.rs gap gap
src/logs/trace.rs gap gap
src/logs/trace/ancestry.rs gap gap
src/logs/trace/ancestry/tests.rs gap gap
src/main.rs gap gap
src/notification.rs gap gap
src/pass.rs gap gap
src/pickup.rs gap gap
src/process/tests.rs gap covered
src/proxy.rs gap gap
src/ready.rs gap gap
src/resend_key.rs gap gap
src/run_ending.rs gap gap
src/sandbox.rs gap covered
src/security.rs gap gap
src/security/close.rs gap gap
src/security/reproduction.rs gap gap
src/security/review.rs gap covered
src/session.rs covered gap
src/session/execution_tests.rs gap gap
src/session/kind.rs gap gap
src/session/recording/tests.rs gap covered
src/session/trace.rs gap gap
src/session/trace/calls.rs gap gap
src/session/trace/calls/tests.rs gap gap
src/session/trace/tests.rs gap gap
src/setup.rs gap gap
src/setup/questions.rs gap gap
src/test_support.rs covered gap
src/trial.rs gap gap
src/verdict.rs gap gap
src/verdict/prediction.rs gap gap
src/verdict/tests.rs gap gap
src/worktree/acquisition_tests.rs gap gap
src/worktree/preservation_tests.rs gap gap
src/worktree/review_recovery_tests.rs gap covered
tests/architect_run.rs gap gap
tests/base_fix.rs gap covered
tests/ci.rs gap gap
tests/codex_sessions.rs gap covered
tests/command_logs.rs gap gap
tests/console.rs gap gap
tests/console/accounts.rs gap gap
tests/console/dismiss.rs gap gap
tests/console/experiments.rs gap gap
tests/console/github.rs gap gap
tests/console/idle.rs gap gap
tests/console/inbox.rs gap gap
tests/console/password_codes.rs gap gap
tests/console/resources.rs covered gap
tests/console/session-expiry.py gap gap
tests/console/spans.rs gap gap
tests/console/watching.rs gap gap
tests/console/waterfall.rs gap gap
tests/convert_logs.rs gap gap
tests/dismiss.rs gap gap
tests/email_test.rs gap gap
tests/eval_arms.rs gap gap
tests/evals.rs gap gap
tests/experiments.rs gap gap
tests/experiments/model_graders.rs gap gap
tests/experiments/rechecks.rs gap gap
tests/experiments/release.rs gap gap
tests/experiments/sandbox.rs gap gap
tests/export_trace.rs gap gap
tests/failed_run.rs gap covered
tests/fixtures/console-claude.jsonl gap covered
tests/fixtures/console-codex.jsonl gap covered
tests/fresh_run.rs gap gap
tests/mergeable.rs gap covered
tests/opencode_sessions.rs gap gap
tests/pickup_run.rs gap gap
tests/progress_lines.rs gap gap
tests/release.rs gap gap
tests/release/experiments.rs gap gap
tests/resources.rs gap gap
tests/run_health.rs gap covered
tests/run_notification.rs gap gap
tests/scoring.rs covered gap
tests/security_review.rs gap gap
tests/security_run.rs gap gap
tests/session_traces.rs gap gap
tests/setup.rs gap gap
tests/show_logs.rs gap gap
tests/spec_run.rs gap gap
tests/status.rs gap gap
tests/support/mod.rs gap covered
tests/support/session_trace.rs gap gap
tests/trial.rs gap gap
tests/user_config.rs gap gap

Merge Danger

Door: two-way

Private proposals can be regenerated and do not alter Run metrics. No destructive migration.

Blast Radius: factory

The feature changes Scoring and shared private Grader sessions. The dependency merge also brings main's Console, Experiment, resource and log changes into the Spec branch; review gaps are disclosed above.

Built with codex · gpt-6.1-sol · xhigh

JacobStephens2 and others added 30 commits October 23, 2026 20:18
The Inbox row's state word is now computed once, beside the grouping: a
Needs you kind, then Stopping for a live Command given a Stop, Ready and
Landing for those groups, otherwise the Command's own state. The Console
row's data-state and the status text line both use it, so every Theme's
Landing and Ready tag colours and the Stopping stripe apply. status --json
adds stopping beside landing. Landing keeps Working's live-activity
marker and a Stopping row keeps Working's tag colours.
Link the GitHub Issue URL from Stages and Command details
Review follow-ups: InboxItem::state_word names the row's word apart
from the Command's own state, live_record takes its issue number, and
the manual says only Command record rows carry the word in status text.
Fix Stages rows to follow Ticket run progression
A Pickup run takes the lowest-numbered Ready issue by default. Set
pickup.order to highest-first in the User config, or pass lowest-first
or highest-first on the command, to choose which end of the issue
numbers it takes from. Two schedules working through the same queue
from opposite ends collide less often; the origin reservation still
protects either choice.

Closes #1109
Jacob agreed to all nine recommendations, noting that the Inbox now works
more like a mail inbox: Needs you and Ready for review keep a row until
it is dealt with, and age clears only what GitHub can't speak for (Q2.5).
ADR 0023 records that the Console reads GitHub once a minute and why;
ADR 0022 was taken on main meanwhile. ADR 0016 points to it. The glossary's
Dismissal, Inbox, Needs you and Activity log entries now say what brings a
dismissed row back, that a dismissal can be taken back, and how GitHub
settles a row.

Round 3 asks what happens when GitHub can't be read, how the reads give
way to the factory's quota, how long a row is read for, where a row that
leaves late goes, whether GitHub moves live rows, where the Console offers
Dismiss, settings, what a waiting row shows of GitHub, dismissing several
rows, the "(title unavailable)" rows, and how #919 is published.
Give Inbox rows in Landing, Ready for review and Stopping their state
Command details now has a Copy CLI button beside its thirdshift show
command. When the clipboard is refused, the command is selected so it
can be copied with the keyboard.
A live Command refreshes Command details on each change, which cleared
the by-hand selection a refused clipboard leaves. The refresh now keeps
the copy status and selection. The copy handler keys on data-copy-cli.
Jacob agreed to all eleven recommendations, and will publish the Spec and
its Tickets with to-spec and to-tickets (Q3.11). ADR 0023 now says what
the Inbox does when GitHub can't be read, how the Console leaves the last
tenth of the quota to the factory, how long a row is read for, that
GitHub moves only ended rows, that Resolved counts from when a row
stopped waiting, and that none of it is a setting. The glossary's Inbox
entry says Resolved's new clock.

Round 4 asks whether to merge main into this branch, take its Ticket run
wording, and land the docs before the Spec, and whether a row that ages
out still skips Resolved.
…g out still skips Resolved

Jacob agreed to both recommendations. A row that ages out isn't one anybody dealt with, so it still goes straight to History (Q4.2); the glossary's Inbox entry and ADR 0023 say so. Q4.1 merges main into this branch next, takes its Ticket run wording, and opens a docs-only pull request before the Spec is written.
The CI fmt check flagged unformatted lines in the new pickup order code.
This is a formatting-only change with no behavior change.
A dispatched run is now the Spec run or Ticket run an Architect run or Security run dispatched, and a Pickup run is skipped while another Pass, or the run it dispatched, is still running on its repository.
Record #919's grilling: Dismissal, and the Inbox reading GitHub (ADR 0023)
JacobStephens2 and others added 25 commits October 24, 2026 12:36
Let Stages fill the window's full width
Add live Experiment pages to the Console
Count same-second Claim hand-backs in either label order
Approve proposed Eval tasks from the CLI and Console
Preserve both the Trial-input preflight documentation and the Console Experiment page documentation on the homepage.
Keep the Trial-input preflight and Spec-review sandbox documentation alongside the incoming CLI and Console Eval-task approval documentation.

Validation: cargo check, cargo fmt --all -- --check, 44 site tests, three Console Experiment browser tests, and all 171 tests in evals, experiments, trial and console passed (three skipped).

The affected-tests selector stopped after 400 Rust passes on release::an_interrupted_release_resumes_the_same_experiment, matching the intermittent failure documented in upstream PR #1273. Its isolated cargo test and nextest reruns both passed, and it passed again in the 171-test targeted run. The selector took 432.74s including compilation; the targeted run took 190.67s.
Complete Spec-review Trial inputs before Experiment builds
Preserve Experiments alongside main's trace recording, Inbox evaluation, resource admission and Console views. Keep older Arms readable and route Experiment release messages through the shared timestamp helper.
Spec: Experiments, which measure a change before it reaches production
A Bug-origin session can replace proposal.json with a symlink. Scoring read that path on the host after the sandbox exited, so the link could import a host file into the saved proposal or block on a fifo while the machine-wide Scoring lock was held. The model Grader read scores.json the same way. Both reads now accept only a regular file with one name and a bounded size.
Combine production Grader checkpoints and carry-over with Bug-origin proposals. Preserve newer process supervision, native quota observation, Session traces and bounded session-output reads.

Retain the incoming sandbox secret isolation, explicitly publish private Grader packet paths, and scope the Scoring fake-launcher override so Experiment and Trial checks still exercise real sandboxes. Regenerate prompts and documentation and retain a combined origin/Grader resume regression.

Validation: formatting and clippy pass; all 50 site checks and all Rust test targets pass across the affected-test run and completion run. The release-resume and worker-overlap Experiment tests each passed exact reruns after intermittent failures during parallel runs.
# Conflicts:
#	README.md
#	docs/code-navigation.md
#	docs/guide.md
#	site/architecture/index.html
#	site/docs/index.html
#	site/how-it-works/index.html
#	site/index.html
#	src/main.rs
#	src/sandbox.rs
#	src/scoring.rs
@JacobStephens2
JacobStephens2 merged commit d27c133 into issue-978 Oct 24, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1028 branch October 24, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant