Repository navigation
feat: @github/copilot-sdk Compatibility Testing & Breaking Change Detection — PRD #818
Description
Problem Statement
Squad has zero runtime compatibility testing against @github/copilot-sdk. All 150+ test files mock the SDK, so interface changes, removed methods, renamed properties, and import path changes are invisible until users update and hit runtime errors. The postinstall ESM patches (vscode-jsonrpc exports + copilot-sdk session.js) are brittle — they depend on exact file paths and string patterns with no verification they actually work.
Current State
What EXISTS
- ✅ Unit tests with full mock coverage (~150 files)
- ✅ Export smoke test (validates 70+ subpath exports load)
- ✅ Postinstall ESM patches (Layer 1: vscode-jsonrpc exports, Layer 2: session.js import rewrite)
- ✅ Sample build validation (15+ samples compile against squad-sdk)
What's MISSING
- ❌ No Dependabot/Renovate — no automated dependency update PRs
- ❌ No nightly/weekly integration tests against SDK pre-releases
- ❌ No unmocked SDK tests — every test mocks CopilotClient
- ❌ No postinstall patch verification — patches fail silently
- ❌ No canary/insiders channel for pre-release SDK testing
- ❌ No breaking change detection before user impact
Breaking Change Detection Timeline (Current)
| Stage | Detection? | When User Learns |
|---|---|---|
| SDK pre-release | ❌ Not tested | — |
| PR merge | ❌ Mocks pass | — |
| npm publish | ❌ Exports valid | — |
| User runs npm update | 💥 Runtime crash | Too late |
Requirements
FR-1: Dependency Update Automation (Dependabot)
Automated PRs when copilot-sdk publishes new versions.
Acceptance criteria:
-
.github/dependabot.ymlconfigured for daily npm updates - copilot-sdk updates create PRs automatically
- CI gates prevent merge of breaking SDK versions
- Ignore pre-releases by default (tested separately via canary)
FR-2: Nightly Canary Workflow
Test against copilot-sdk@next (pre-release) every night.
Acceptance criteria:
-
.github/workflows/squad-canary-sdk.ymlruns on schedule (daily 4AM UTC) - Installs copilot-sdk@next, builds squad-sdk, runs tests
- On failure: creates issue or posts comment alerting team
- Manual dispatch supported for ad-hoc testing
- Tests both Node 22 and Node 24
FR-3: Unmocked SDK Smoke Test
At least one test that imports @github/copilot-sdk WITHOUT mocking to validate the real interface.
Acceptance criteria:
- Test imports CopilotClient without vi.mock()
- Validates CopilotClient exists and is a constructor
- Validates expected methods exist on prototype (start, stop, createSession, etc.)
- Validates vscode-jsonrpc/node subpath is importable
- Runs as part of standard CI (squad-ci.yml test job)
- Failure clearly identifies which method/property is missing
FR-4: Postinstall Patch Verification
Defense-in-depth: verify ESM patches actually applied after install.
Acceptance criteria:
-
scripts/verify-patches.mjsruns afterpatch-esm-imports.mjs - Verifies vscode-jsonrpc/package.json has
exports["./node"] - Verifies copilot-sdk session.js uses
.jsextension on imports - Fails install with clear error if patches didn't apply
- Added to postinstall chain in package.json
- CI validates on clean install (rm node_modules → npm install → verify)
FR-5: SDK Version Compatibility Matrix
Test against multiple copilot-sdk minor versions on PRs touching SDK code.
Acceptance criteria:
- Matrix workflow tests 3-4 SDK versions (current, current-1, current-2, next)
- Triggers on changes to packages/squad-sdk/ and packages/squad-cli/
- Reports which versions pass/fail
- Does NOT block merge for next/pre-release failures (informational)
FR-6: Patch Deprecation Tracking
ESM patches are temporary workarounds. Track when they can be removed.
Acceptance criteria:
- Each patch file has a TODO with the upstream issue/PR that will make it unnecessary
- Nightly canary tests with patches removed to detect when upstream fixes land
- When upstream fix detected: auto-create PR to remove patch
Non-Functional Requirements
NFR-1: No CI Slowdown for Happy Path
SDK compatibility checks must not add >30s to normal PR CI. Matrix/canary tests run separately.
NFR-2: Graceful Degradation
If copilot-sdk@next is unavailable (npm outage), canary workflow skips gracefully — no false alarm.
NFR-3: Clear Error Messages
When SDK incompatibility is detected, the error must say exactly:
- Which SDK version broke
- Which method/property/import changed
- Link to the SDK changelog or commit
Design Decisions
DD-1: Keep Caret Range with Canary Safety Net
Keep ^0.1.32 for flexibility. Canary tests provide 1-2 week early warning. Pin only if a specific version is known broken.
DD-2: Smoke Test in Main CI, Matrix in Separate Workflow
The unmocked smoke test runs on every PR (fast, 1 SDK version). The full matrix runs only on SDK-touching PRs or nightly.
DD-3: Postinstall Patches Are Tech Debt
Track patch removal as explicit tech debt with upstream issue links. Goal: zero patches when copilot-sdk ships native ESM.
Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| copilot-sdk ships breaking minor | Medium | High — runtime crash for all users | FR-1 + FR-2 (early detection) |
| Postinstall patch breaks on new SDK | High | Medium — install succeeds but runtime fails | FR-4 (verification) |
| ESM import paths change | Medium | High — silent failure | FR-3 (unmocked test catches) |
| Canary creates noise (flaky failures) | Medium | Low — team ignores alerts | NFR-2 (graceful skip) |
Effort Estimates
| Requirement | Effort | Priority | Dependencies |
|---|---|---|---|
| FR-1: Dependabot config | 1-2 hrs | 🔴 Critical | None |
| FR-2: Nightly canary workflow | 4-6 hrs | 🔴 Critical | None |
| FR-3: Unmocked smoke test | 3-4 hrs | 🟠 High | None |
| FR-4: Patch verification script | 2-3 hrs | 🟠 High | None |
| FR-5: SDK version matrix | 3-4 hrs | 🟡 Medium | FR-3 |
| FR-6: Patch deprecation tracking | 1-2 hrs | 🟡 Medium | None |
| Total | ~15-21 hrs |
Implementation Order
- FR-1 (Dependabot) — zero-code, immediate value
- FR-4 (Patch verification) — catches silent failures now
- FR-3 (Unmocked smoke test) — catches interface breaks
- FR-2 (Nightly canary) — early warning system
- FR-5 (Version matrix) — comprehensive coverage
- FR-6 (Patch deprecation) — tech debt cleanup
References
- Gap analysis from project-dina session (2026-04-04)
- SDK dependency: packages/squad-sdk/package.json → @github/copilot-sdk@^0.1.32
- ESM patches: packages/squad-cli/scripts/patch-esm-imports.mjs
- Doctor checks: packages/squad-cli/src/cli/commands/doctor.ts (lines 294-380)
- Related: Init/Upgrade Parity PRD (separate issue)
Activity
diberry commentedon Apr 6, 2026
Verdict: PRD is complete and ready for implementation.
This is a thorough PRD with FR-1 through FR-6 covering the full dependency update → canary → unmocked smoke → patch verification → matrix → deprecation pipeline. NFRs address false-positive risk, backward compat, and CI performance.
Strengths:
Minor gaps (non-blocking):
Recommendation: Implementable as-is. Suggest adding a Phase 1 (FR-3 + FR-4) → Phase 2 (FR-1 + FR-2) → Phase 3 (FR-5 + FR-6) ordering note.
feat(ci): SDK compatibility testing — dependabot, canary, smoke test,…
feat(cli): init/upgrade parity tests, CI gate, doctor global install UX
diberry commentedon Apr 10, 2026
Closing — descoping high-touch feature work to focus on existing PRs and low-touch fixes.