Skip to content

Freeze Custom process Arms and home skills for Experiment Trials - #1928

Merged
JacobStephens2 merged 5 commits into
issue-1484from
issue-1877
Nov 10, 2026
Merged

JacobStephens2 merged 5 commits into
issue-1484from
issue-1877

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Freeze a Custom process and its referenced home skills for either Arm, retaining each file's SHA-256 and every Step's Harness, Model and Effort. Baseline defaults to the Built-in Ticket run.

prepare Arms → freeze Process + home skills + Step choices
Trial       → verify both Arms → copy into fresh home
            → execute Steps → stop before publishing
mismatch    → stop Experiment before another Trial; name changed file

Closes #1877

Evidence

Execution evidence

  • Before: cargo test --test eval_arms custom_process_and_home_skill_are_frozen_with_step_choices failed with the prepare-arms usage error.
    After: Same command passes (1 test); frozen Process, instructions, helper and Step choices recorded.
  • Before: cargo test --test experiments custom_process_trials_use_frozen_home_skills_beside_the_builtin_baseline failed when a later Trial reread the changed original Process (TOML parse error).
    After: Same command passes: 6 delivered Trials, Custom Candidate Steps and Built-in Baseline Steps.
  • Before: The Spec review's exact commands python3 .thirdshift-review-FzmsRH/repro_spec.py issue and python3 .thirdshift-review-FzmsRH/repro_spec.py sandbox both failed with codex exited 55; the latter reported gitfile does not point to a valid repository.
    After: Both exact commands exit 0 with outcome: delivered. Retained regression cargo test --test trial custom_throwaway_steps_receive_the_issue_and_keep_git_access_on_another_harness -- --nocapture passes both pass-through and bwrap modes.
  • cargo clippy --bin thirdshift -- -D warnings passes. Frozen-file tampering, mixed pinned Harnesses and older Built-in-only Baseline protocol regressions pass.
  • CI is green for the exact head 524206a9: 3,947 Rust tests passed, 91 skipped, plus Site, Console, formatting, all-target clippy and Python tooling. cargo nextest run took 19m45s including 1m03s compilation (1,121.179s test execution). Both Linux release builds also passed.
  • Local scripts/affected-tests origin/issue-1484 selected site checks and the full Rust suite: 68 site checks and 1,663 Rust unit tests passed. The first full run stopped because the host selected Python 3.9 without tomllib; cargo test --test bug_origins settlement::historical_review_and_reclosed_origin_regressions -- --nocapture passed with Python 3.11. The Python 3.11 retry was stopped after successful CI covered the exact head, reusing CI as docs/testing.md instructs.

Test seams: the evals prepare-arms, experiment and hidden --trial CLI interfaces, plus the rendered site documentation's format contract.

Visual evidence

Documentation captures use the same expanded sections and viewport states. Before is the pinned issue-1484 base; After is the implementation. Public documentation only.

Home, Experiment section

1440×900 desktop

Before

After

390×844 mobile

Before

After

How it works, Process Trials

1440×900 desktop

Before

After

390×844 mobile

Before

After

Manual, Process Trials

1440×900 desktop

Before

After

390×844 mobile

Before

After

Architecture, Trial execution

1440×900 desktop

Before

After

390×844 mobile

Before

After

Review coverage

Reviewed against issue-1484 at 8ad2733c with thirdshift-code-review. Standards: 1 finding, addressed. Spec: 2 findings, addressed. Every behavioral reproduction was run as written before deciding. Both axes read all 26 changed files; no unread changed files and no generated-output exclusions. Reports and finding counts are retained in .thirdshift-review-FzmsRH/{standards.md,spec.md,summary.json} beside the Session evidence.

Unaddressed findings

  • Standards: None.
  • Spec: None.
  • Security: None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Blast Radius: Evals

Experiment JSON advances to format 5 to describe frozen Process inputs; older saved Arm records remain readable. Process files needing GitHub before their first publishing Step remain refused.

Built with codex codex-cli 0.166.0 · gpt-6.1-sol · xhigh

A Process Trial keeps one home for every Step and prepares a new sandbox
there when a Step uses another Harness. Copying a login through a symlink
in that home wrote outside the sandbox. Host writes and read-only binds
into the fresh home now refuse a symlink along the destination.
Preserve Custom process and frozen home-skill documentation alongside the process Eval task proposal guidance. Regenerate the manual from the merged source.
Preserve frozen Custom process Arms and home skills alongside final-commit process grading with the reference tests and shared fakes overlaid. Regenerate the manual and reconcile the three hand-written site pages.

Validation: cargo check --bin thirdshift, cargo fmt --check, all-target clippy, and all 68 site checks pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant