Skip to content

COMPILE_CMD/TEST_CMD assume a runnable test suite exists #56

Closed

Description

@dmccoystephenson

What

The generated-skill template uses {{COMPILE_CMD}} and {{TEST_CMD}} as if every repo has a runnable build and test command:

  • create-dev-loop.md:262-263 — the Phase 3 "Verify the build is clean" fenced block
  • create-dev-loop.md:406 — Phase 6, "Run {{TEST_CMD}} after every fix. Do not push a broken build."
  • create-dev-loop.md:530 — the Phase 8 rebase edge case

Their Step 4 substitution-table rows offer no guidance for a repo that has neither:

Row Current text
COMPILE_CMD "Fastest command that catches syntax/import errors without running tests. For Maven: …"
TEST_CMD "Full test suite command from CI workflow or README. For Maven: $MVN test …"

Compare LINT_CMD ("omit section if absent") and EXTERNAL_SIGNAL_CMD (explicitly handles the manual-validation case) — both tell the generator what to do when the thing doesn't exist. COMPILE_CMD and TEST_CMD don't.

Why it matters

create-dev-loop itself is such a repo — CLAUDE.md states "There is no automated test suite" — and so is any docs-only or config-only repo. The generator currently has to improvise: the existing create-dev-loop-dev-loop skill renders Phase 6 as "Re-run the Phase 3 manual validation checklist after every fix" and the build block as a grep placeholder-consistency check with a parenthetical noting no test command exists. That improvisation works, but it is re-derived per repo instead of being encoded, which is exactly the drift the template is supposed to prevent.

PR #28 fixed this class of problem for the self-review anchor by introducing EXTERNAL_SIGNAL_LABEL / EXTERNAL_SIGNAL_CMD. The build/test commands were not given the same treatment.

Suggested fix

Give both rows explicit no-build-system / no-test-suite guidance, and make the Phase 3 block and the Phase 6 / Phase 8 sentences degrade gracefully (e.g. conditional blocks, or substitute the project's validation checklist the same way EXTERNAL_SIGNAL_CMD already does). Keep phase numbers stable and add table rows for any new placeholder.

Research grounding

RESEARCH.md §8 (prompt/model drift): forward-only template fixes leave existing skills silently lagging, and a rule re-derived per generation rather than encoded is the same failure mode by a different route. No finding speaks directly to build-command modeling — flagging that honestly rather than reaching for one.

Provenance

Found during a Stage A documentation-accuracy sweep; deferred out of that cycle's PR because fixing it properly is a template design change (likely a new conditional or placeholder), not doc drift.


drafted by Claude on behalf of Daniel Stephenson

Activity

  1. added a commit that references this issue on Aug 3, 2026
    85f80e7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions