Skip to content

[1/3] docs+test(squad-2.0): Squad feature x host matrix with live Claude/OpenCode runs and bridge tests - #2277

Merged
tamirdresher merged 5 commits into
devfrom
squad2-a-host-feature-matrix
Oct 23, 2026
Merged

tamirdresher merged 5 commits into
devfrom
squad2-a-host-feature-matrix

Conversation

@tamirdresher

@tamirdresher tamirdresher commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

Layer 1/3 of the Squad 2.0 pre-preview follow-up stack (supersedes #2274 once the stack is registered). Ships Data's Squad-feature × host matrix as a tested artifact: test-fixtures/spec/suite-v0.2/host-feature-matrix.json (source of truth), scripts/render-host-feature-matrix.mjs, the rendered reference page, and test/squad-20-host-feature-matrix.test.ts. Plus matrix-only doc corrections (F2 GitLab "planned, not shipped", F3 ADO scope note, F5 headless Claude --mcp-config). No product code.

Why

The matrix is the evidence base for Layers 2 and 3: every feature × host cell carries a capability (supported|degraded|unsupported|n/a) and an independent validation level (validated-live|validated-test|static-only|untested|blocked) backed by an evidence[] list (validated-test:<file>::<name>, validated-manual:lab:<scenario>@<date>, static-only) and the live-run log (Claude Code 2.1.296 + OpenCode across 15 scenarios; Kiro/Antigravity/Cursor/Codex blocked by quota/auth on the reference machine). Findings F1–F9 are recorded here as unsupported digest-gap cells (pinned by per-host it.fails) or degraded cells with a stated limitation, so the later layers flip them with evidence.

How

  • Cherry-picked Data's two commits from tamirdresher-microsoft-squad2-host-feature-matrix and squashed with the follow-ups below.
  • Renderer: escapes | in feature names (markdownlint MD056 was red on Data's branch) and compares line-ending-insensitively in --check (autocrlf checkouts).
  • Satellite fixture uses '../base' (the Windows separator failed lstat on Linux CI).
  • Expected-fail digest gaps are expectedFailure: true cells with capability: unsupported (never degraded or supported) and validation: validated-test (the it.fails proves the gap), listed in an expectedFailures block rendered as "Expected failures (it.fails)" (Q item 5 + taxonomy redline).
  • cspell words for the live-run names; "behaviour" → "behavior".
  • Docs: scenarios/using-squad-across-clients.md (F5: claude -p … --mcp-config .mcp.json; Squad does not project enableAllProjectMcpServers), features/enterprise-platforms.md (F3: ADO = detection + squad triage|watch via az; interactive Issues mode GitHub-only), features/gitlab-issues.md + tour-gitlab-issues.md (F2: planned, not shipped).

Review items → evidence (this layer)

Item Evidence
Matrix integrity test/squad-20-host-feature-matrix.test.ts › "host-feature-matrix.json integrity" (every validated-test ref resolves, every lab ref has a live run, page in sync, runtime flags ⇔ capability tokens)
Expected failures labelled (Q-5) expectedFailures block + expectedFailure: true / unsupported on each gap cell; rendered section; integrity test "expected-failure cells are unsupported…"
FUP-10 CRLF-safe --check scripts/render-host-feature-matrix.mjs --check normalises CRLF
Linux CI fixture satellite teamRoot: '../base'

⚠️ Quick Check

  • No SDK/CLI source files changed → no changeset required (gate covers src/templates only)

PR Readiness Checklist

Branch & Commit

  • Branch created from dev (0446cd7 on 510aa25)
  • Branch is up to date with dev
  • Verified diff contains only intended changes (11 files)
  • PR is not in draft mode
  • Commit history is clean (1 commit)

Build & Test

  • SKIP_BUILD_BUMP=1 npm run build — passed (product code = dev)
  • npm run lint — passed
  • npm run lint:eslint — 0 errors (pre-existing warnings only)
  • npm run lint:docs — passed (markdownlint 0, cspell 0)
  • npx vitest run test/squad-20-host-feature-matrix.test.ts test/clients.test.ts test/template-sync.test.ts test/ux-gates.test.ts test/cli-command-wiring.test.ts test/docs-build.test.ts --testTimeout=180000 — 594 passed / 7 expected-fail / 0 failed after rebuilding on this layer (the earlier ux-gates/Thread 6 misses were the 5 s spawn window on a host running 600+ node processes; both pass alone)
  • npm test full — CI requested as the authoritative run (host saturated)

Changeset

  • N/A — no packages/*/src or template changes

Docs

  • Feature pages updated (matrix reference page; F2/F3/F5 notes)

Exports

  • N/A

Breaking Changes

None.

Waivers

None requested.


Reviewer appendix — exact mapping 0446cd71 → this head

Schema

  • v1 cell {status, validation[]} → v2 cell {capability, validation, evidence[], blockedReason?, expectedFailure?}. capability ∈ supported|degraded|unsupported|n/a; validation ∈ validated-live|validated-test|static-only|untested|blocked. Old statuses/validationKinds → capabilities/validationLevels/evidenceKinds (all defined on the page). The old failing marker is gone; digest gaps are expectedFailure: true + capability: unsupported.
  • Derivation (deterministic, matrix-v2 migration): expected-failure feature ⇒ unsupported + validated-test (the per-host it.fails proves the gap); else a passing validated-manual:lab:* run for that host ⇒ validated-live; else any validated-test: ⇒ validated-test; else old untested + a blocked live run for the host ⇒ blocked(reason); else old untested ⇒ untested; else static-only. Capability otherwise unchanged from v1 status.

Features

  • removed: consult-personal-squad (retired; row, gap test, expected failure and rendered section all purged — git grep -i consult test-fixtures/spec docs/src/content/docs/reference/host-feature-matrix.md test/clients.test.ts scripts matches only two live-run notes that use the ordinary verb "consulted")

Capability changes (cells whose v1 status ≠ v2 capability)

Feature Host v1 status v2 capability v2 validation Why
model-selection copilot supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection claude degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection opencode degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
model-selection antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues copilot degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues claude degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues opencode degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
gitlab-issues antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops copilot degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops claude degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops opencode degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
azure-devops antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance copilot degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance claude degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance opencode degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
upstream-inheritance antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root copilot supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root claude supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root opencode supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
monorepo-team-root antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode copilot supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode claude supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode opencode supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
dual-root-remote-mode antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins copilot supported unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins claude degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins kiro degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins opencode degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins cursor degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins codex degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)
marketplace-plugins antigravity degraded unsupported validated-test digest gap pinned by per-host it.fails ⇒ unsupported (Q: a missing coordinator section is not a degradation)

49 cells changed capability; every other cell keeps its v1 status as capability.

Validation levels per host

Host validated-live validated-test static-only untested blocked
copilot 0 49 2 0 0
claude 18 30 3 0 0
kiro 0 47 3 0 1
opencode 18 30 3 0 0
cursor 0 47 3 0 1
codex 0 47 3 1 0
antigravity 0 47 3 0 1

validated-live appears only for claude and opencode (enforced by the integrity test). blocked = cursor/kiro/antigravity cells whose only evidence was static and whose live smoke run was blocked by a usage limit on 2026-10-22.

Expected failures (expectedFailures[], exact titles)

  • monorepo-team-root → test/clients.test.ts::digest gap (expected until the stack adds it): team-root resolution (teamRoot / stateLocation / TEAM_ROOT) + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • dual-root-remote-mode → test/clients.test.ts::digest gap (expected until the stack adds it): team-root resolution (teamRoot / stateLocation / TEAM_ROOT) + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • upstream-inheritance → test/clients.test.ts::digest gap (expected until the stack adds it): upstream inheritance (upstream.json) + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • azure-devops → test/clients.test.ts::digest gap (expected until the stack adds it): Azure DevOps issue tracking (az boards / ADO) + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • gitlab-issues → test/clients.test.ts::digest gap (expected until the stack adds it): GitLab issue tracking (glab) + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • model-selection → test/clients.test.ts::digest gap (expected until the stack adds it): per-agent model / reasoning / context-tier selection + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity
  • marketplace-plugins → test/clients.test.ts::digest gap (expected until the stack adds it): plugin marketplace browse on cast + [<host>] for copilot, claude, kiro, opencode, cursor, codex, antigravity

Generated in test/clients.test.ts as it.fails(`digest gap (expected until the stack adds it): ${name} [${id}]`) for every id in CLIENT_IDS (6 gaps × 7 hosts = 42). The integrity test resolves each entry against the gap literal, the generating template and the CLIENT_IDS loop, and requires every listed cell to be expectedFailure: true + unsupported, and every flagged cell to be listed.

validated-test references (61 distinct; each file exists and each non-* name appears literally in it — asserted by "every validated-test reference names a test that exists")

  • validated-test:test/aspire-command.test.ts::*
  • validated-test:test/cli/doctor.test.ts::*
  • validated-test:test/cli/export-import.test.ts::*
  • validated-test:test/cli/init.test.ts::*
  • validated-test:test/cli/rc.test.ts::*
  • validated-test:test/cli/upgrade.test.ts::*
  • validated-test:test/clients-real-cli.test.ts::antigravity discovers the squad agent
  • validated-test:test/clients-real-cli.test.ts::claude discovers the squad subagents and parses .mcp.json
  • validated-test:test/clients-real-cli.test.ts::codex loads the project config (trusted), sees squad_state MCP, discovers the squad TOML role, and activates coordinator instructions on spawn
  • validated-test:test/clients-real-cli.test.ts::cursor-agent lists the squad_state server from .cursor/mcp.json (read-only)
  • validated-test:test/clients-real-cli.test.ts::kiro validates and lists the squad agent
  • validated-test:test/clients-real-cli.test.ts::opencode loads squad as a primary agent and connects the MCP server
  • validated-test:test/clients.test.ts::Reviewer gate (@Bubu): /client <-> /clients export identity (===), pure projection boundary, and byte-exact rollback across all 7 hosts
  • validated-test:test/clients.test.ts::coordinator Ralph mode
  • validated-test:test/clients.test.ts::coordinator behavioral-contract parity
  • validated-test:test/clients.test.ts::coordinator digest vs. host feature matrix
  • validated-test:test/clients.test.ts::portable clients[] hygiene
  • validated-test:test/cross-squad-registry.test.ts::*
  • validated-test:test/cross-squad.test.ts::*
  • validated-test:test/gh-aw-enlistment-skill.test.ts::*
  • validated-test:test/memory-governance.test.ts::*
  • validated-test:test/migrate.test.ts::*
  • validated-test:test/model-selector-policy.test.ts::*
  • validated-test:test/otel-export.test.ts::*
  • validated-test:test/platform-adapter.test.ts::*
  • validated-test:test/ralph-board.test.ts::*
  • validated-test:test/ralph-commands.test.ts::*
  • validated-test:test/ralph-monitor.test.ts::*
  • validated-test:test/ralph-triage.test.ts::*
  • validated-test:test/rate-limiting.test.ts::*
  • validated-test:test/resolution.test.ts::*
  • validated-test:test/skills.test.ts::*
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::12. capability tokens match capability-matrix.json for all 7 hosts
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::R3-9: createMcpBackend honors a configured (non-local) state backend instead of writing .squad/backend
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::RC1→stable exit 1: upgrade and rollback from a v0.1 workspace
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 1: client projections and --client
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 2: cross-plane capability intersection
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle
  • validated-test:test/squad-20-all-scenarios.e2e.test.ts::field findings from the live demo: Init Mode, scratch files, casting registry
  • validated-test:test/squad-20-host-feature-matrix.test.ts::claude --agent squad reads the roster through mcp__squad_state__squad_state_read
  • validated-test:test/squad-20-host-feature-matrix.test.ts::claude lists the root .mcp.json squad_state server from packages/api
  • validated-test:test/squad-20-host-feature-matrix.test.ts::opencode --agent squad reads the roster through squad_state_squad_state_read
  • validated-test:test/squad-20-host-feature-matrix.test.ts::opencode resolves the squad primary agent and the squad_state MCP entry from packages/api
  • validated-test:test/squad-20-host-feature-matrix.test.ts::squad_decide over the bridge lands in the squad-state branch for the %s backend, not the working tree
  • validated-test:test/squad-20-host-feature-matrix.test.ts::state-mcp follows config.json teamRoot from a satellite repo (remote mode)
  • validated-test:test/squad-20-host-feature-matrix.test.ts::state-mcp resolves the root .squad when launched from a monorepo subfolder
  • validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in
  • validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::executes squad_state tools against the workspace .squad directory
  • validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::negotiates capabilities for the client per capability-matrix.json
  • validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::writes the host config and agent files each client reads
  • validated-test:test/state-backend.test.ts::*
  • validated-test:test/team-capabilities-lifecycle.test.ts::*
  • validated-test:test/upgrade-state-backend.test.ts::*
  • validated-test:test/upstream-e2e.test.ts::*
  • validated-test:test/upstream.test.ts::*
  • validated-test:test/watch-fleet-dispatch.test.ts::*
  • validated-test:test/watch-notify-level.test.ts::*
  • validated-test:test/watch-rate-limit.test.ts::*
  • validated-test:test/watch-subsquad-discovery.test.ts::*
  • validated-test:test/worktree.test.ts::*

Renderer

  • node scripts/render-host-feature-matrix.mjs --check compares LF- and CRLF-normalized content (autocrlf checkouts pass).
  • Page sections: Legend (capability / validation / evidence definitions), Summary (two independent breakdowns + expected-failure count), Expected failures, Known limitations, Matrix (capability validation glyph pairs), Per-feature detail (Capability | Validation | Mechanism | Evidence), Live run log.

Copilot threads on #2277 (all resolved with commit + test)

  1. per-host it.fails (thread 1) → test/clients.test.ts gap loop over CLIENT_IDS.
  2. EF refs validated (thread 2) → "every expected failure resolves to one it.fails title per listed host…".
  3. failing ⇒ not supported (thread 3) → now "expected-failure cells are unsupported…" (stronger: unsupported, not merely not-supported).
  4. live-model tests assert bridge calls (thread 4) → tee'd squad state-mcp transcript, successfulRosterReads() with its own unit test; the page states this is the only bridge-level assertion.

Copilot AI balanced review requested due to automatic review settings October 22, 2026 15:40
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit 86402cc

PR Scope: 🔧 Infrastructure

⚠️ 2 item(s) to address before review

Status Check Details
❌ Single commit 5 commits — consider squashing before review
✅ Not in draft Ready for review
✅ Branch up to date Up to date with dev
❌ Copilot review No Copilot review yet — it may still be processing
✅ Changeset present No source files changed — changeset not required
✅ Scope clean No .squad/ or docs/proposals/ files
✅ No merge conflicts No merge conflicts
✅ Copilot threads resolved 0 active Copilot thread(s) resolved (4 outdated skipped)
✅ CI passing All checks passing

Files Changed (11 files, +6882 −3)

File +/−
cspell.json +2 −0
docs/src/content/docs/features/enterprise-platforms.md +2 −0
docs/src/content/docs/features/gitlab-issues.md +1 −1
docs/src/content/docs/reference/host-feature-matrix.md +1056 −0
docs/src/content/docs/scenarios/using-squad-across-clients.md +1 −1
docs/src/content/docs/tour-gitlab-issues.md +1 −1
docs/src/navigation.ts +1 −0
scripts/render-host-feature-matrix.mjs +206 −0
test-fixtures/spec/suite-v0.2/host-feature-matrix.json +5082 −0
test/clients.test.ts +39 −0
test/squad-20-host-feature-matrix.test.ts +491 −0

Total: +6882 −3


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

@github-actions

Copy link
Copy Markdown
Contributor

🟡 Impact Analysis — PR #2277

Risk tier: 🟡 MEDIUM

📊 Summary

Metric Count
Files changed 11
Files added 4
Files modified 7
Files deleted 0
Modules touched 4

🎯 Risk Factors

  • 11 files changed (6-20 → MEDIUM)
  • 4 modules touched (2-4 → MEDIUM)

📦 Modules Affected

docs (6 files)
  • docs/src/content/docs/features/enterprise-platforms.md
  • docs/src/content/docs/features/gitlab-issues.md
  • docs/src/content/docs/reference/host-feature-matrix.md
  • docs/src/content/docs/scenarios/using-squad-across-clients.md
  • docs/src/content/docs/tour-gitlab-issues.md
  • docs/src/navigation.ts
root (2 files)
  • cspell.json
  • test-fixtures/spec/suite-v0.2/host-feature-matrix.json
scripts (1 file)
  • scripts/render-host-feature-matrix.mjs
tests (2 files)
  • test/clients.test.ts
  • test/squad-20-host-feature-matrix.test.ts

This report is generated automatically for every PR. See #733 for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The evidence checks can accept missing bridge calls, miss individual host transitions, and allow inconsistent matrix statuses and test references.

4 open issues
What changed in this PR

Establishes the documentation and test evidence baseline for the Squad 2.0 follow-up stack, without changing product code.

Changes:

  • Adds a generated host-feature reference page with validation evidence and recorded live-run outcomes.
  • Adds matrix-integrity checks, bridge scenarios, opt-in host tests, and expected-failure digest checks.
  • Clarifies GitLab availability, Azure DevOps support scope, and headless Claude configuration.
File Description
test/​squad-20-host-feature-matrix.test.ts Adds integrity, bridge, and opt-in host checks.
test/​clients.test.ts Records digest coverage and expected gaps.
scripts/​render-host-feature-matrix.mjs Renders the matrix and checks synchronization.
docs/​src/​navigation.ts Adds the matrix to reference navigation.
docs/​src/​content/​docs/​tour-gitlab-issues.md Marks the walkthrough as planned functionality.
docs/​src/​content/​docs/​scenarios/​using-squad-across-clients.md Clarifies headless Claude MCP setup.
docs/​src/​content/​docs/​reference/​host-feature-matrix.md Publishes feature support and validation evidence.
docs/​src/​content/​docs/​features/​gitlab-issues.md Clarifies that GitLab integration is not shipped.
docs/​src/​content/​docs/​features/​enterprise-platforms.md Distinguishes Azure DevOps CLI and interactive support.
cspell.json Adds names used in the evidence documentation.

Review effort: Balanced

Comment thread test/clients.test.ts Resolved
Copilot and others added 5 commits October 23, 2026 07:51
…enCode runs and bridge tests [1/3]

Adds test-fixtures/spec/suite-v0.2/host-feature-matrix.json (source of truth),
scripts/render-host-feature-matrix.mjs, the rendered reference page, and
test/squad-20-host-feature-matrix.test.ts (schema/evidence integrity, rendered
page in sync, runtime flags vs capability tokens, deterministic bridge
scenarios: monorepo subfolder, remote-mode satellite, orphan/two-layer
squad_decide; F6 pinned as it.fails). Live validation: Claude Code 2.1.296 +
OpenCode across 15 scenarios; Kiro/Antigravity/Cursor/Codex blocked
(quota/auth) and kept static-only. Findings F1-F9 recorded as degraded cells
and it.fails digest gaps in test/clients.test.ts for Layers 2/3 to flip.

Follow-ups folded in (no product code):
- renderer escapes `|` in feature names (markdownlint) and compares
  line-ending-insensitively in --check; cspell words for live-run names;
  "behaviour" -> "behavior".
- the satellite fixture uses '../base' (the Windows separator failed lstat
  on Linux CI).
- expected-fail digest gaps are labelled `failing` with a note on every
  degraded cell and listed in an `expectedFailures` block rendered as
  "Expected failures (it.fails)" — never counted as supported.
- docs: headless Claude needs `claude -p … --mcp-config .mcp.json`
  (F5; Squad does not project enableAllProjectMcpServers), ADO support =
  detection + `squad triage|watch` via `az`, interactive Issues mode is
  GitHub-only (F3), GitLab Issues pages marked planned, not shipped (F2).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b822a455-fa94-49a2-9127-eee479ab954e
…ells, tee'd bridge evidence [1/3]

Review follow-ups on the host feature matrix (Worf LA-2..7, Copilot threads 1-4):

- expectedFailures now resolve to the exact generated it.fails titles: one test per
  host and gap (`digest gap (expected until the stack adds it): <gap> [<host>]`);
  the integrity suite validates every entry against test/clients.test.ts, requires
  every listed cell to be `failing` and not `supported`, and every `failing` cell to
  be covered by an expected failure.
- Cells that were `supported` while carrying an expected-fail digest gap (team-root
  on copilot/claude/opencode, model-selection and marketplace on copilot) are now
  `degraded` with the gap test attached; `failing` cells can never be `supported`.
- Retired `consult-personal-squad` row, gap test and expected failure removed.
- Cursor: installed on the reference machine (live run blocked by a usage limit);
  its evidence kinds follow the no-live-run rule (`untested`), like kiro/codex/antigravity.
- Live-model tests assert a successful `squad_state_read(team.md)` tools/call in a
  tee'd `squad state-mcp` transcript instead of stdout substrings; parser unit-tested.
- Page gains "Known limitations of this evidence".

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b822a455-fa94-49a2-9127-eee479ab954e
…expected-failure integrity check [1/3]

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b822a455-fa94-49a2-9127-eee479ab954e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b822a455-fa94-49a2-9127-eee479ab954e
…ps are unsupported [1/3]

Q's taxonomy redline: capability (what the host can do) and validation confidence (how
we know) are independent axes.

- Cell schema v2: `capability: supported|degraded|unsupported|n/a` and
  `validation: validated-live|validated-test|static-only|untested|blocked` with
  `evidence[]` (test refs, recorded live runs, static marker), `blockedReason`, and
  `expectedFailure` for digest gaps. The v1 `failing` marker is gone.
- Every feature whose coordinator section is absent from a host digest is `unsupported`
  for that host (49 cells; previously degraded/supported) — pinned per host by it.fails,
  so validation is `validated-test`. 12 cells stay `degraded` with a stated limitation.
- `validated-live` only for Claude Code and OpenCode (passing recorded runs); cursor,
  kiro and antigravity cells with static-only evidence and a blocked live attempt are
  `blocked(reason)`.
- All eight terms (4 capability + 5 validation minus overlap) defined on the page;
  summary shows both breakdowns; per-feature detail shows both axes plus evidence.
- "through the bridge" wording in live-run notes replaced by "host-reported squad_state
  tool events"; the page states the tee'd JSON-RPC transcript check is the only
  bridge-level assertion.
- Integrity tests: axes enum-checked and cross-checked against evidence kinds
  (validated-live needs a lab entry and a live host; validated-test needs a test ref;
  static/untested/blocked must not list test/live evidence; blocked needs a reason;
  n/a is static-only); expected-failure cells must be unsupported and cite the gap test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b822a455-fa94-49a2-9127-eee479ab954e
@tamirdresher
tamirdresher force-pushed the squad2-a-host-feature-matrix branch from 33f3408 to 86402cc Compare October 23, 2026 05:02
@tamirdresher
tamirdresher merged commit 894e3c4 into dev Oct 23, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants