Skip to content

gh-aw quality suite silently skips 13 tests (28 assertions) on Windows via existsSync('/bin/sh') probe #1833

Closed

Description

@bradygaster

Summary

test/gh-aw-quality.test.ts:30 gates two describe blocks on a POSIX shell that never exists on Windows:

const HAS_POSIX_SHELL = existsSync('/bin/sh');
// :1515
describe.skipIf(!HAS_POSIX_SHELL)('gh-aw: Team Guard roster-row detection — committed-HEAD git repo coverage (#1689 revision 3)', …)
// :1687
describe.skipIf(!HAS_POSIX_SHELL)('gh-aw: Cast PR dedup jq filter behavioral coverage (#1689 revision)', …)

On Windows, /bin/sh does not exist, so both blocks skip. Measured: 13 tests containing 28 expect() calls never execute — and the suite reports green.

This is the same defect class as #1824 / #1812 / #1801 / #1822 / #1827: a gate that structurally cannot observe the failure it exists to catch. Per the standing decision in .squad/decisions.md (2026-08-20): a permanently green — or permanently red — gate is equivalent to no gate.

Why this one stings

The skipped blocks are #1689's own behavioral coverage — assertions written specifically to prove gh-aw Team Guard and Cast PR dedup behavior. A Windows contributor sees a green suite and reasonably concludes that behavior is verified. It isn't.

Git Bash ships a POSIX shell on essentially every Windows dev machine that has Git installed, so the skip isn't even buying portability — it's discarding coverage that could have run.

Prior art in this repo — the fix shape already exists

test/gh-aw-command-parse.test.ts (added in #1832 for #1824) hit the identical problem and solved it: it resolves Git Bash rather than probing /bin/sh, and then hard-asserts that a shell was found. If no shell can be resolved, the suite goes red, not green.

FIDO verified this by force during review of #1832 — forcing the no-shell path produced 13 RED, a loud guard failure rather than a silent skip.

That is the correct shape and it is already in the tree. This issue is to apply it to gh-aw-quality.test.ts.

Proposed fix

  1. Replace the existsSync('/bin/sh') probe with the Git Bash resolution helper used by gh-aw-command-parse.test.ts (extract it to a shared test util rather than copying — a third parallel implementation is how this class of bug spreads).
  2. Remove skipIf entirely. Hard-assert a shell was resolved so absence reports red.
  3. Confirm the 13 tests actually pass on Windows once they run. They may not — they have never executed in this environment, so treat any failure as a real, previously-hidden defect rather than a porting nuisance.

Acceptance criteria

  1. gh-aw-quality.test.ts executes all 13 previously-skipped tests on Windows, or fails loudly if no POSIX shell can be resolved. No silent skip path remains.
  2. Proven capable of failing: with the shell resolution forced to fail, the suite reports red and the message names the missing dependency. A status-only assertion is not sufficient — per fix(gh-aw): fail loudly when /squad parses no command (#1824) #1832's finding, a generic-diagnostic mutation passes a status-only check clean.
  3. Reported test count for the file increases by 13 on Windows. State the before/after counts as measured, not estimated.
  4. No remaining skipIf-on-environment-probe in the gh-aw test suites — or, for any that remain, an explicit comment justifying why absence is genuinely not a failure.

Provenance

Found by Procedures during #1824 (PR #1832), out of scope there and deliberately left alone. Original report said "102 skipped assertions"; FIDO re-measured it as 13 tests / 28 expects — a counting-convention discrepancy, not a mechanism error. The mechanism is confirmed real. Numbers in this issue are FIDO's measured figures.

Activity

  1. added
    bugSomething isn't working
    gh-awGitHub Agentic Workflows integration
    triagedIssue has been triaged
    wave:1-nextWave 1 — immediate next sprint
    on Aug 23, 2026
  2. added a commit that references this issue on Aug 25, 2026
    0b5d83a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggh-awGitHub Agentic Workflows integrationpriority:p1This sprinttriagedIssue has been triagedwave:1-nextWave 1 — immediate next sprint

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions