Skip to content

Stop prescribing commands that headless harnesses refuse - #176

Merged
dmccoystephenson merged 2 commits into
mainfrom
fix/refused-command-guidance
Nov 10, 2026
Merged

dmccoystephenson merged 2 commits into
mainfrom
fix/refused-command-guidance

Conversation

@dmccoystephenson

Copy link
Copy Markdown
Member

Summary

Commands that harnesses refuse were prescribed, or left unmentioned, in two parts of the generated-skill template. Both are corrected in the template body of create-dev-loop.md.

Closes #135
Closes #165

Deviations from #135, based on source checks made in this dispatch:

  • It is said in the issue that gh pr view --comments "returns the same information" as the gh api call. That was not confirmed. When it was run against PR Make Steps 6 and 7 safe to re-run under overwrite mode #175, conversation comments and the Copilot review summary were shown, but no inline comment text was shown.
  • It is implied in the issue that a --jq template without a literal Body: label is accepted. In this dispatch, gh api repos/Stephenson-Software/create-dev-loop/pulls/175/comments --jq '.[] | "\(.path):\(.line) — \(.body)"' was still refused with This command requires approval. The call is therefore written as optional, not as the fix.

gardener interface: Steps 3, 5, and 6 are untouched, so no breaking change is introduced for gardener.

Retrofit: the Phase 5 snippet and the Phase 3 rule are copied into every generated skill, so a retrofit pass is likely needed for every existing <slug>-dev-loop skill. Per #135, Phase 5 was already fixed locally in medieval-factions-dev-loop. The old snippet is still carried by create-dev-loop-dev-loop.

Research grounding

No RESEARCH.md finding applies. These corrections are based on refused commands that were observed, and no empirical claim is relied on, so no Implementations entry is added.

Doc sync check

  • README.md's "What it does" Step list still matches create-dev-loop.md's Steps 1:1 (no Step was changed)
  • Every {{placeholder}} added or changed has a corresponding Step 4 substitution-table row (none were added; GITHUB_OWNER and GITHUB_REPO are unchanged)

Test plan

  • python3 scripts/check_docs.py: "Doc consistency check passed."
  • python3 -m unittest discover -s tests: 42 tests were run, OK.
  • gh pr view 175 --comments was run, and the claims made in Phase 5 were checked against its output.
  • The gh api read was attempted in this dispatch and was refused, as warned in Phase 5. Its output format has therefore not been observed.
  • The jq-filter pattern from Phase 3 was run: both PRs were returned by gh pr list --state merged --json number,mergedAt -q '.[]|select(.number|IN(174,175))'.
  • UNVERIFIED (behavioral): no /create-dev-loop run was performed, because a GitHub repo is created in Step 6 and files outside the checkout are written in Steps 3 and 5, and neither is authorized in a headless dispatch ([self-audit] The repo's own validation anchor is structurally unrunnable in a headless dispatch, so template changes can never merge autonomously #109). None of the five CLAUDE.md checklist items was exercised, so this PR is left for a maintainer to merge.

The template body is not covered by anything automated, so no evidence about generated-skill behavior is given by a green doc-consistency job. The loop refusal named in #165 was not re-run here. It is taken from the observations recorded in #165 and from a later dispatch's notes.

Issues not picked this cycle

This PR description was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).

🤖 Generated with Claude Code


drafted by Claude on behalf of Daniel Stephenson

Phase 5 now reads reviews through gh pr view and treats the inline-comment
gh api read as optional, since it can be refused even though it only reads.
A refusal or an empty result is no longer readable as "no review comments".

Phase 3's command-substitution rule now also covers shell variable loops and
$-anchored sed/awk, with a single jq-filtered gh call as the replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dmccoystephenson

Copy link
Copy Markdown
Member Author

Self-review rubric (head a682682). This comment is not an independent review; it was produced by the session in which the PR was authored.

Universal items:

Repo-specific items:

  • Placeholder parity: PASS — no placeholder was added, and check_docs.py passed. GITHUB_OWNER and GITHUB_REPO are still used, so their rows are not orphaned.
  • README Step parity: PASS — no Step was added, removed, or renamed.
  • Fence escaping: PASS — one line was changed inside an already-escaped fence, and no fence was added.
  • Phase numbering: PASS — no phase was renumbered.
  • Version comments: PASS (not applicable) — the header was not touched.
  • Research grounding: PASS — it is stated in the PR body that no finding applies.
  • No back-ported specifics: PASS — only {{GITHUB_OWNER}}/{{GITHUB_REPO}} and generic commands appear in the added text.
  • gardener interface: PASS — Steps 3, 5, and 6 were not changed.
  • Retrofit note: PASS — it is stated in the PR body that every existing generated skill likely needs a pass.
  • Behavioral validation: FAIL — marked UNVERIFIED in the PR body, so no auto-merge.

Points for the maintainer's review, none of which were changed:

Summary: the change reads correctly and CI is green, but the two FAIL items hold it for a maintainer, so it is not being merged by this dispatch.

This comment was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The review flow lacks a blocking fallback when review summaries are insufficient and inline comments are inaccessible, allowing progression toward merge without the findings Phase 6 needs.

1 open issue
What changed in this PR

This PR updates the generated dev-loop template to reduce command refusals in headless harnesses, addressing the command patterns reported in #135 and #165.

Changes:

  • Expands Phase 3 guidance to avoid shell-variable loops and favor a single filtered gh call.
  • Uses gh pr view commands as Phase 5 defaults, retaining the inline-comment API read as optional and documenting the defaults’ limitations.
File Description
create-dev-loop.md Revises Phase 3 command guidance and Phase 5 review retrieval instructions for restricted harnesses.

Review effort: Balanced

Comment thread create-dev-loop.md Outdated
gh pr view <number> --json reviews
\`\`\`

These show conversation comments and each review's state and summary, but not the text or `path:line` anchor of inline review comments. Where the summary names the findings, Phase 6 can work from it. Where the harness allows it, this read fills the gap: `gh api repos/{{GITHUB_OWNER}}/{{GITHUB_REPO}}/pulls/<number>/comments --jq '.[] | "\(.path):\(.line) — \(.body)"'`. A command allow-list or attribution hook can refuse it even though it only reads. **A refusal, or an empty `--comments` result, is never evidence that no review comments exist** — check `--json reviews` for a submitted review before concluding there is nothing to address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dmccoystephenson

Copy link
Copy Markdown
Member Author

The Copilot finding on a682682 was assessed against the source and was found valid. It was read from the review summary, since the inline-comment read was refused in this dispatch, so its title was used and its line anchor was not seen.

  • create-dev-loop.md:424 — "Block merge when review findings cannot be retrieved": fixed. Under the earlier text, a review whose findings could be read in neither its summary nor its inline comments could have been passed over on the way to merge. A hold and a hand-off are now required for that case.

The self-review rubric was not re-run, as Phase 4 caps it at one pass. python3 scripts/check_docs.py passed and 42 unit tests were run, OK, locally on the fix. The behavioral check remains UNVERIFIED, and the PR stays held for a maintainer.

This comment was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

@dmccoystephenson
dmccoystephenson merged commit 855a4b7 into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants