Skip to content

fix: 35 failing tests on dev blocking all PR CI checks #220

New issue
New issue
Closed
Closed
fix: 35 failing tests on dev blocking all PR CI checks#220
Labels
regressionTest regression or broken featurerelease:nextTargeted for next release

Description

@williamhallatt
williamhallatt
opened on Mar 6, 2026

Problem

The dev branch has 35 failing tests across 11 test files, which means CI fails on every PR that targets dev. This has been the case for at least the last 5 commits (since 8b33e6d — Phase 1 SDK-First tests).

Verified locally by checking out upstream/dev and running npm test.

Failing test files (11)

test/acceptance/acceptance.test.ts         (2 failures)
test/acceptance/hostile.test.ts            (2 failures)
test/cli-p0-regressions.test.ts            (6 failures)
test/cli/init.test.ts                      (1 failure)
test/docs-build.test.ts                    (1 failure)
test/first-run-gating.test.ts              (2 failures)
test/human-journeys.test.ts                (5 failures)
test/repl-ux-e2e.test.ts                   (5 failures)
test/repl-ux-fixes.test.ts                 (1 failure)
test/repl-ux.test.ts                       (5 failures)
test/ux-gates.test.ts                      (5 failures)

Total: 35 failed | 3582 passed | 25 skipped

Root cause clusters

1. --version outputs prefixed string (7 tests)

Tests expect bare semver (0.8.21-preview.4) but CLI outputs squad 0.8.21-preview.4.
Affects: cli-p0-regressions, human-journeys, repl-ux-e2e, ux-gates

2. Empty/whitespace args exit with code 1 instead of 0 (5 tests)

Tests expect help display + exit 0 but CLI exits 1.
Affects: cli-p0-regressions, human-journeys, acceptance/hostile

3. Missing squad doctor hint in error messages (4 tests)

Unknown command errors don't include remediation hint.
Affects: cli-p0-regressions, human-journeys, ux-gates

4. Help output structure (2 tests)

Tests expect grouped commands and footer with squad <command> --help / Getting Started.
Affects: ux-gates

5. REPL shell requires interactive TTY (4 tests)

squad shell exits with ✗ Squad shell requires an interactive terminal in test environment instead of showing welcome/init flow.
Affects: repl-ux-e2e

6. TUI component regressions (5 tests)

  • AgentPanel shows [IDLE] but tests expect idle status text
  • Animations: ✓ Done flash not rendering (shows ✓ [IDLE])
  • Keyboard: Backspace not deleting last character
  • Welcome: agent roster not showing team members
    Affects: repl-ux

7. init output text changed (1 test)

Test expects Scaffold ready but output is Let's build your team.
Affects: human-journeys

8. .gitattributes path mismatch (1 test)

Test expects .squad/decisions/decisions.md merge=union but actual output differs.
Affects: cli/init

9. runInit missing directory/file creation (1 test)

runInit doesn't create all required .squad/ directories and structural files.
Affects: repl-ux-fixes

10. NODE_NO_WARNINGS not set before imports (1 test)

cli-entry.ts doesn't set NODE_NO_WARNINGS=1 as first statement.
Affects: first-run-gating

11. Banner double-spacer (1 test)

Banner has more than one spacer between banner and version line.
Affects: first-run-gating

12. Docs build failure (1 test)

docs-build.test.ts failing.
Affects: docs-build

13. Acceptance: consult + status commands (2 tests)

Affects: acceptance/acceptance

Impact

All PRs targeting dev show ❌ CI — contributors cannot get a green build. PRs #217 and #219 are both affected.

Activity

  1. williamhallatt commented on Mar 6, 2026

    @williamhallatt
    williamhallatt
    on Mar 6, 2026
    ContributorAuthor

    Root cause investigation findings

    Running the test suite locally on upstream/dev confirms 35 failing tests across 11 files.

    Confirmed: Implementation regressions (17/35)

    Root commit: 4ecc244 — "feat: Squad Remote Control"
    This commit rewrote large portions of cli-entry.ts and silently dropped multiple P0 bug fixes that had been deliberately committed in earlier fixes:

    Regression 1: --version outputs squad X.X.X instead of bare semver (7 tests)

    • Bug was fixed in 5664fd0 (CLI polish) and 00d2713 (P0 bugs [A2A] Discovery Mechanism (Local + Network) #333): console.log(VERSION)
    • 4ecc244 rewrote the file and reintroduced console.log(squad ${VERSION})
    • Tests in: cli-p0-regressions, human-journeys, repl-ux-e2e, ux-gates

    Regression 2: Empty/whitespace args exit code 1 instead of 0 (5 tests)

    • Bug was fixed in 00d2713: added rawCmd?.trim() guard + whitespace-only → help + exit 0
    • 4ecc244 reverted back to const cmd = args[0] without the trimming
    • Tests in: cli-p0-regressions, human-journeys, acceptance/hostile

    Regression 3: NODE_NO_WARNINGS=1 not set before imports (1 test)

    Regression 4: REPL first-run (no .squad/) shows TTY error (4 tests)

    • The TTY guard in runShell() fires unconditionally when not in a TTY
    • Tests in repl-ux-e2e expect first-run (no .squad/) to output a welcome message and squad init hint even in non-interactive contexts (e.g., piped)
    • Tests were written alongside the TTY guard — the intended design was that first-run should degrade gracefully (show help + exit 0) rather than erroring

    Still investigating (18/35)

    9 additional clusters are being analyzed to determine if they're further regressions or aspirational tests:

    • squad doctor hint in error messages (4 tests)
    • Help output grouping/structure (2 tests)
    • TUI regressions: AgentPanel status, animations, keyboard (5 tests)
    • init output text (1 test)
    • .gitattributes merge paths (1 test)
    • runInit missing directories (1 test)
    • Banner spacer (1 test)
    • Docs build (1 test)
    • Acceptance tests — consult + status (2 tests)

    Will update this issue with full classification once the second wave of investigation completes.

    Investigation from fork: williamhallatt/fix-dev-failing-tests

  2. williamhallatt commented on Mar 6, 2026

    @williamhallatt
    williamhallatt
    on Mar 6, 2026
    ContributorAuthor

    Complete root cause classification — all 35 failures

    Investigation complete. 2 tests turned out to be false positives (they actually pass — Backspace and docs-build). Real failure count: 33 tests.

    REGRESSIONS — implementation needs fixing (25 tests)

    Previously confirmed (17 tests):

    • Regression 1: --version outputs squad X.X.X instead of bare semver (7 tests) — 4ecc244 dropped fix from 5664fd0/00d2713
    • Regression 2: Empty/whitespace args exit code 1 instead of 0 (5 tests) — 4ecc244 dropped fix from 00d2713
    • Regression 3: NODE_NO_WARNINGS=1 not set (1 test) — 4ecc244 dropped fix from ceb599b
    • Regression 4: REPL first-run TTY guard fires unconditionally (4 tests) — design intent was graceful first-run degradation

    Newly confirmed (8 tests):

    Regression 5: squad doctor hint missing from unknown-command errors (4 tests)

    • Test files: cli-p0-regressions, human-journeys, ux-gates
    • Root commit: 00d2713 added the two-hint message: "Run 'squad doctor' to check your setup, or 'squad help' for usage information."
    • It was restored in 0e08594, then silently dropped again in 4ecc244
    • Fix: restore line 305 in cli-entry.ts

    Regression 6: Completion flash animation broken — shows ✓ [IDLE] instead of ✓ Done (3 tests)

    • Test file: repl-ux.test.ts
    • Root commit: b0c3a7b added useCompletionFlash() hook and ✓ Done render on working→idle transition
    • Commit 72ffcb1 (unified status display) introduced getStatusTag('idle') → '[IDLE]' which overrode the flash text
    • Fix: restore flash conditional in AgentPanel.tsx so ✓ Done renders during flash window instead of [IDLE]

    Regression 7: Banner spacer source test regex broken (1 test)

    • Test file: first-run-gating.test.ts
    • Commit d1c73c0 (REPL UX round 2) refactored headerElement from inline JSX useMemo(() => (...), []) to useMemo(() => { if (narrow) ... }, [])
    • Test regex expects the old inline arrow-function form — now produces null match
    • Fix: update regex in first-run-gating.test.ts to match new function body pattern

    TEST-WRONG — test expectations are incorrect (6 tests)

    These tests have expectations that never matched the implementation, or were invalidated by a deliberate refactor:

    Test file Line Wrong expectation Correct expectation
    repl-ux.test.ts 154 toContain('ready') (name says "idle status") toContain('[idle]') (the actual tag)
    human-journeys.test.ts 137 toContain('Scaffold ready') toContain('Your team is ready') (never matched impl)
    cli/init.test.ts 94 .squad/decisions/decisions.md (wrong path) .squad/decisions.md (canonical path per SDK impl)
    consult-command.feature 25 "already has a .squad/" "No personal squad found" (personal squad checked first)
    status-extended.feature 15 exit code 1 for non-squad dir exit code 0 (informational, not an error — f8b91d1 changed this incorrectly)

    ASPIRATIONAL — desired behavior not yet implemented (2 tests)

    Cluster B: Grouped help output (ux-gates.test.ts lines 76-95)

    • Tests expect commands grouped into categories ("Getting Started", "Development", etc.) with a per-command help footer
    • No implementation of grouped help has ever existed — cli-entry.ts has always used a flat list
    • Test added in 9f044f3 (accessibility audit) as forward-looking spec
    • Needs decision: implement grouped help, or update tests to match current flat format

    FALSE POSITIVES — actually passing (2 tests)

    • repl-ux.test.ts: "Backspace deletes the last character" — passes
    • docs-build.test.ts: All 26 tests pass — docs/build.js was fixed since original investigation

    Summary

    Category Tests Action
    Regressions 25 Fix implementation
    Test-wrong 6 Fix tests
    Aspirational 2 Design decision needed
    False positive 2 No action
    Total reported 35

    All regressions trace back to two commits: 4ecc244 (Squad Remote Control) and 72ffcb1 (unified status display) — both dropped or overrode previously working behavior without reverting the associated tests.

    Fix branch: williamhallatt/fix-dev-failing-tests

  3. williamhallatt
    added a commit that references this issue on Mar 6, 2026

    fix: restore CI green on dev — 25 regressions + 6 test corrections

    67b3578

  4. williamhallatt commented on Mar 6, 2026

    @williamhallatt
    williamhallatt
    on Mar 6, 2026
    ContributorAuthor

    Additional finding: Playwright CI gap

    While investigating the 35 failing tests, found one more pre-existing failure in test/aspire-integration.test.ts:

    Error: browserType.launch: Executable doesn't exist at
  /home/runner/.cache/ms-playwright/chromium_headless_shell-1208/...

    Root cause: Commit fea6a67 (feat: Wave 2 — Aspire Playwright E2E) added the Aspire integration test but didn't add a npx playwright install step to the CI workflow. The test has a Docker-based skip guard, but GitHub Actions ubuntu-latest runners have Docker — so the test runs and fails immediately because Chromium isn't installed.

    Fix: Added to PR #221 — one new step in .github/workflows/squad-ci.yml:

    - name: Install Playwright browsers
  run: npx playwright install chromium --with-deps

    This is the standard Playwright CI pattern from their own docs. Adds ~30s to CI runs.

  5. williamhallatt
    added a commit that references this issue on Mar 6, 2026

    ci: add npx playwright install chromium step to squad-ci.yml

    7f37881

  6. bradygaster commented on Mar 6, 2026

    @bradygaster
    bradygaster
    on Mar 6, 2026
    Collaborator

    amazing! file a pull request

  7. bradygaster
    added
    release:nextTargeted for next release
    regressionTest regression or broken feature
    on Mar 6, 2026

  8. williamhallatt commented on Mar 6, 2026

    @williamhallatt
    williamhallatt
    on Mar 6, 2026
    ContributorAuthor

    PR was up yesterday along with the issue. It's linked in the last comment before yours 😄

    Here again for convenience:

    https://github.com/bradygaster/squad/pull/221

  9. bradygaster commented on Mar 7, 2026

    @bradygaster
    bradygaster
    on Mar 7, 2026
    Collaborator

    ✅ Fixed by PR #221 (merged commit 0315468) — 25 test regressions resolved.

    Closed as part of v0.8.21 release wave cleanup.

  10. bradygaster
    closed this as completedon Mar 7, 2026
  11. jongio
    added 2 commits that reference this issue on Mar 9, 2026

    feat: CRLF normalization utility + apply to all 8 parsers

    6ca8ac2

    docs(ai-team): Epic github#181 completion — session log + decision merge

    94ba535
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    regressionTest regression or broken featurerelease:nextTargeted for next release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions