Skip to content

Measure skill catalogs and clarify Factory resolution - #764

Merged
JacobStephens2 merged 4 commits into
mainfrom
issue-735
Oct 17, 2026
Merged

JacobStephens2 merged 4 commits into
mainfrom
issue-735

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Closes #735.

Record evidence-backed skill catalogs and recommend keeping Run startup unchanged. Preserve two concurrent investigations as separate observations; their different roots and counts have no established causal explanation. No harmful Factory skill selection was demonstrated in either bounded sample.

Actual Codex catalogs       fresh    continued   reviewers
This session                47/32    59/44       61/46
Concurrent investigation    65/51    50/36       65/51 (two)
                            entries / distinct skill names
                            15 versus 14 duplicated names

The notes explain namespaced Factory resolution, supported Codex/Claude configuration, intentionally available tools, and Continuation/Resume limits. Separate JSON baselines retain each investigation's provenance. The concurrent Continuation is an older 0.162.1 observation; these counts are not controlled comparisons of session types or CLI versions.

Evidence

  • Before: Measure redundant skill catalogs and clarify Factory skill resolution #735's retrospective supplied a historical 44-skill count without a current catalog baseline.
    After: Retain seven actual session observations across the two investigations. Both published recomputation commands pass, matching counts, bytes, proxy tokens, duplicates and SHA-256 hashes. Different session IDs/catalog roots remain explicitly attributed rather than averaged or interpreted as a change rate.
  • Before: The fresh catalog contained 47 entries / 13,277 bytes / 3,159 proxy tokens.
    After: An invocation-only, no-turn diagnostic suppressed 15 duplicate Host entries: 32 entries / 9,506 bytes / 2,251 proxy tokens, retaining all ten Factory names and accountability instructions. Personal configuration stayed byte-identical. This measures catalog savings; it does not establish billed savings, runtime selection, hook execution or Resume compatibility. No startup suppression is shipped.
  • Review: Standards: 0 findings. Spec: 1 P3 UTC-label finding, reproduced with the reviewer's exact command, corrected, and verified with the unchanged command. The check is retained in the research note. Fresh independent Standards and Spec passes cover the combined branch.
  • Validation: cargo check, cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test --test continuation, and cargo test skills::tests::links_factory_skills_in_existing_and_missing_directories_for_every_adapter passed. Full cargo test: 1,872 passed, 2 ignored, 0 failed across 48 test binaries. The subsequent reconciliation changed only documentation/evidence; Rust source, skills and tests are unchanged. Typechecking, formatting, both catalog recomputations and the retained UTC check pass after reconciliation.

Validation seams: model-visible Codex prompt input and recorded catalogs; existing Factory-skill worktree linking and Continuation CLI flow; displayed UTC timestamps against captured metadata. No new production behavior was introduced for this investigation.

Unaddressed findings

  • Standards: None.
  • Spec: None; the P3 finding was addressed.

Both reviewers read all five changed files on the combined branch. Unread changed files: none. Initial and combined reports, including final files-read lists, are saved in the requested .thirdshift-review-df7VoO/standards.md and spec.md files in the worktree.

The earlier guidance-mode Security review reported no introduced findings at concurrent commit 6006f33; that review predates the documentation reconciliation and is not claimed as a Security audit of the combined head.

Security review outcome

  • Security review refused: Codex's cybersecurity safeguard refusal

Merge Danger

Door: two-way

Documentation and a captured baseline only; no Harness settings, embedded skills, instruction files, hooks or personal/global configuration changes.

Blast Radius: documentation

Catalog measurements are host/version/context snapshots. Proxy token counts do not prove end-to-end cost or latency improvements.

Built with codex · gpt-6.1-sol · high

CI notes

  • test-macos-shard (2), run 39054889637: all 448 tests passed; GitHub marked the job cancelled when newer PR run 39056268502 superseded it (annotation: "Canceling since a higher priority waiting request for CI-pull_request-764- exists").
  • test-macos-shard (4), run 39054889637: the same concurrency cancellation interrupted testing; no test failure was reported.
  • test-macos, run 39054889637: the aggregate check correctly failed on RESULT: cancelled; this was caused by the superseded run, not a branch regression, so no code or check changes were made.
  • Local verification on Linux at 970689f: cargo nextest run --partition hash:2/4 passed 463/463 tests; cargo nextest run --partition hash:4/4 passed 484/484 tests. Tests outside each hash partition were excluded by the unchanged partition selection, as in CI; no checks or assertions were changed. macOS validation belongs to replacement run 39056268502.

@JacobStephens2
JacobStephens2 marked this pull request as ready for review October 17, 2026 17:53
@JacobStephens2 JacobStephens2 changed the title Measure skill catalogs and document Factory resolution Measure skill catalogs and clarify Factory resolution Oct 17, 2026
@JacobStephens2
JacobStephens2 merged commit c7ec7ee into main Oct 17, 2026
7 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-735 branch October 17, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure redundant skill catalogs and clarify Factory skill resolution

1 participant