PRD: CI Template Testing Infrastructure — Structural Tests & Pre-Release Validation #888

Problem Statement

Squad ships 11 workflow templates to user repos. PR #887 added caching, timeouts, concurrency, SHA pinning, and security hardening — but there are zero automated tests ensuring these properties persist. A SHA transcription error was caught only by manual security review. Future edits could silently regress any of these properties.

Current State

• test/workflows.test.cjs exists — tests name:, on:, jobs: keys and framework workflow copying on init. Does NOT test structural properties.
• test/template-sync.test.ts exists — 146 tests ensuring byte-identity across 4 template directories. Works great for sync, but doesn't validate content correctness.
• test/ci-concurrency.test.cjs exists — tests Squad's OWN .github/workflows/ CI, NOT the shipped user templates.
• Zero tests for: timeout presence, SHA pinning, concurrency blocks, cancel-in-progress values, cache configuration, bot guards, persist-credentials, id-token scoping, permissions blocks.

Proposed Solution — Two Layers

Phase 1: Structural Tests (must ship)

Add to new test/template-ci-hardening.test.cjs or extend test/workflows.test.cjs:

Tests to write (20 structural assertions):

1. timeout-minutes required — Every template has timeout-minutes on every job
2. SHA pinning (40 hex) — No uses: actions/*@v[0-9]+ pattern. All action refs must be 40-character lowercase hex SHA (e.g., action@abc123...def), NOT floating refs like @main, @latest, @HEAD, @master
3. Concurrency blocks — Every template has a concurrency: block (all 11 templates)
4. cancel-in-progress: false group — Release/promote/insider-release have cancel-in-progress: false (includes: squad-release.yml, squad-promote.yml, squad-insider-release.yml)
5. cancel-in-progress: true group — CI/preview/sync-labels have cancel-in-progress: true (includes: squad-ci.yml, squad-preview.yml, squad-sync-labels.yml, squad-docs.yml, squad-triage.yml, squad-issue-assign.yml, squad-label-enforce.yml, squad-heartbeat.yml)
6. npm cache configuration — Templates with setup-node have cache: npm
7. no npm cache when no setup-node — Templates WITHOUT setup-node do NOT have cache: npm
8. Bot guards on label-triggered workflows — Triage, issue-assign, and label-enforce workflows include bot guards to skip github-actions[bot] and dependabot[bot] runs
9. persist-credentials: false on read-only — Read-only workflows (triage, issue-assign, label-enforce) have persist-credentials: false
10. id-token scoping (indentation-aware) — squad-docs.yml has id-token: write only in the deploy job, NOT at workflow level. Must use indentation-aware regex or YAML parser to distinguish workflow-level vs job-level scoping (simple includes() check is insufficient)
11. SHA format validation — All SHA pins are exactly 40 lowercase hexadecimal characters
12. SHA version comments — All SHA pins have version comments (e.g., # v6.0.2) for audit trail
13. SHA consistency — Same action used across different templates must have identical SHA
14. dorny/paths-filter wiring — squad-ci.yml uses dorny/paths-filter (not paths-ignore). Assertion checks: (a) uses: dorny/paths-filter present, (b) filters: block exists, (c) downstream job references needs.changes.outputs
15. permissions block required — Every template declares a top-level permissions: block (zero-trust baseline — without it, GITHUB_TOKEN defaults to contents: write)
16. no pull_request_target — Shipped templates must not use pull_request_target trigger. Exception: if used, must pair with fork-safety guard github.event.pull_request.head.repo.full_name == github.repository
17. no workflow-level id-token: write — id-token: write must not appear at workflow level in any template (only squad-docs.yml deploy job should have it; all other templates must have no id-token grant or job-level only)
18. no secrets: inherit — Templates must not use secrets: inherit in reusable workflow calls (violates least-privilege principle)
19. HAS_COPILOT_TOKEN pattern — Heartbeat env block correctly uses secrets.COPILOT_ASSIGN_TOKEN != '' boolean comparison, not raw secret interpolation in strings
20. Concurrency group sanity — Every concurrency: expression's group: is non-empty and contains ${{ ... }} interpolation (exception: squad-docs.yml uses static group: pages)

Also added to Phase 1:

• Known-good SHA map — Add test/fixtures/known-good-shas.json. Maps action@tag → expected SHA. Tests compare against this map. Any SHA change requires deliberate fixture update = audit trail.
• YAML schema validation — Use ajv + GitHub workflow JSON schema. Catches structural errors text-matching misses. Validates all 11 templates against https://json.schemastore.org/github-workflow.json

Test infrastructure:

• Use getAllTemplateWorkflows() helper (already exists in workflows.test.cjs) as iteration basis, so new templates are auto-covered by all assertions
• File: test/template-ci-hardening.test.cjs (separate from workflows.test.cjs which tests CLI behavior)
• Dependencies: node:fs, node:assert, ajv (for schema validation)

Estimated effort: 3-4 hours. Zero build-breaking dependencies.

Phase 2: Pre-Release Smoke Testing with act (nice to have)

• Install nektos/act as optional dev documentation
• Create minimal test fixture repo in test/fixtures/act-test-repo/ with package.json + package-lock.json
• Run each template through act with:
  • Happy path: npm project with lockfile
  • Edge case: no lockfile (should degrade gracefully)
  • Edge case: pnpm project (should not hard-fail)
• Document in docs/contributing/testing-workflows.md

SHA resolution companion script — Document in Phase 2: a 20-line shell script using gh api /repos/{owner}/{action}/git/commits/{sha} to verify SHAs resolve. Not CI-blocking, but pre-release manual check.

Estimated effort: 4-6 hours. Requires Docker (manual gate, not CI).

Phase 3: Dynamic Template Enumeration & Broader Coverage (nice to have)

• All structural tests use getAllTemplateWorkflows() for auto-discovery
• Broader floating ref regex catches @main, @latest, @HEAD, @master, not just @vN

Success Criteria

• 20 structural test assertions covering all properties from PRD: CI Template Overhaul — Performance & Safety for Shipped Workflows #886 + team review feedback
• Tests run in existing npm test — no new CI jobs
• Tests catch: missing timeout, floating action tag, wrong cancel-in-progress, missing bot guard, missing permissions block, id-token leaks, secrets: inherit
• Tests fail if a future edit removes SHA pinning or reverts to @v4
• Known-good SHA map prevents silent SHA drift
• YAML schema validation in Phase 1 (promoted from Phase 2)
• Documentation for optional act smoke testing

Non-Goals

• Testing workflow LOGIC (triage routing, label assignment, etc.) — that's a separate concern
• Adding act to CI (too heavy — Docker dependency)
• Testing Squad-internal workflows (.github/workflows/) — covered by ci-concurrency.test.cjs

Team Review History

v2 — Updated with team review from Booster, FIDO, RETRO

Changes from v1:

1. Assertion 3 fix — Removed "(except squad-docs.yml)" exception. ALL 11 templates now confirmed to have concurrency: blocks.
2. Assertions 4/5 expansion — Added 5 missing templates with correct cancel-in-progress grouping:
   • cancel-in-progress: true: squad-docs.yml, squad-triage.yml, squad-issue-assign.yml, squad-label-enforce.yml, squad-heartbeat.yml
   • cancel-in-progress: false: squad-release.yml, squad-promote.yml, squad-insider-release.yml
3. Assertion 10 refinement — Clarified that includes() is insufficient. Must use indentation-aware regex or YAML parser to distinguish workflow-level vs job-level id-token scoping.
4. New assertions 15-18 — Added four critical security assertions:
   • 15: Explicit permissions: block required at top-level (zero-trust baseline)
   • 16: No pull_request_target trigger (or with fork-safety guard)
   • 17: No id-token: write at workflow level (only squad-docs deploy job)
   • 18: No secrets: inherit in reusable workflow calls
5. Assertion 19 — Added HAS_COPILOT_TOKEN pattern validation (boolean comparison, not raw interpolation)
6. Assertion 20 — Added concurrency group sanity check (non-empty, interpolation required except squad-docs static group)
7. Known-good SHA map — Promoted to Phase 1. Maps action@tag → expected SHA with audit trail.
8. YAML schema validation — Promoted to Phase 1 (from "nice to have"). Catches structural errors.
9. dorny/paths-filter wiring — Enhanced Assertion 14 to validate not just presence but also filters: block and downstream needs.changes.outputs reference.
10. Broader floating ref coverage — Assertion 2 now catches @main, @latest, @HEAD, @master and all non-SHA refs.
11. Dynamic template enumeration — Tests use getAllTemplateWorkflows() helper for auto-discovery of new templates.
12. SHA resolution script — Documented as Phase 2 manual pre-release gate.
13. New test file — test/template-ci-hardening.test.cjs separates CI hardening tests from workflow sync/initialization tests.
14. Total structural assertions: 20 (up from 14) + concurrency group sanity.

References

• Issue PRD: CI Template Overhaul — Performance & Safety for Shipped Workflows #886 — PRD: CI Template Overhaul
• PR feat(ci): harden shipped workflow templates — caching, timeouts, concurrency, SHA pins #887 — Implementation (all 3 phases)
• Booster's review: clarified scope for all 11 templates
• FIDO's review: identified zero structural tests as a blocker
• RETRO's review: caught invalid SHA — structural test would have prevented this
• nektos/act — https://github.com/nektos/act
• GitHub workflow JSON schema — https://json.schemastore.org/github-workflow.json