Repository navigation
feat(security): add markdown-aware skill security scanner (Phase 1) - #882
Conversation
🟡 Impact Analysis — PR #882Risk tier: 🟡 MEDIUM 📊 Summary
🎯 Risk Factors
📦 Modules Affectedroot (1 file)
scripts (1 file)
tests (2 files)
This report is generated automatically for every PR. See #733 for details. |
There was a problem hiding this comment.
Pull request overview
Adds Phase 1 markdown-aware security scanning for skill files to the existing scripts/security-review.mjs repo-health scanner, with accompanying test coverage and a changeset.
Changes:
- Extend
security-review.mjsto scan changed skill markdown files under.copilot/skills/**and.squad/skills/**for Tier 1 patterns (credentials, download-and-exec, privilege escalation). - Export a pure
scanSkillContent()function and guard script execution so it can be imported safely from tests. - Add a dedicated Vitest suite for skill scanning behavior and register the new finding categories in the existing schema test.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/security-review.mjs |
Adds markdown-aware scanning (scanSkillContent) and integrates it into the main security-review run. |
test/scripts/security-review.test.ts |
Extends the allowed category allowlist to include new skill-* categories. |
test/scripts/security-review-skills.test.ts |
New comprehensive unit tests for skill scanning patterns + suppression behavior + golden corpus. |
.changeset/skill-security-scanner.md |
Declares a patch release for the CLI package to ship the new scanner behavior. |
14d1572 to
4076c6d
Compare
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | Changeset file found |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ❌ | Copilot threads resolved | 2 unresolved Copilot thread(s) — fix and resolve before merging |
| ❌ | CI passing | 1 check(s) still running |
| ✅ | Issue linked | Issue reference found |
| ✅ | Protected files | No protected bootstrap files changed |
Files Changed (4 files, +866 −0)
| File | +/− |
|---|---|
.changeset/skill-security-scanner.md |
+5 −0 |
scripts/security-review.mjs |
+206 −0 |
test/scripts/security-review-skills.test.ts |
+651 −0 |
test/scripts/security-review.test.ts |
+4 −0 |
Total: +866 −0
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Extends security-review.mjs with Tier 1 pattern detection for skill markdown files (.copilot/skills/**/*.md and .squad/skills/**/*.md): - P1: 10 embedded credential patterns (AWS, GitHub, OpenAI, JWT, etc.) - P2: 7 credential file read patterns (.env, .ssh/id_rsa, .aws/credentials, .npmrc, .netrc) - P3: 8 download-and-execute patterns (curl|bash, irm|iex, etc.) - P4: 5 privilege escalation patterns (sudo, RunAs, chmod 777, etc.) Suppression (Phase 1): - Fenced code blocks (CommonMark-compliant: backtick/tilde, variable length) - Inline code spans (backtick pairs) - Regex-doc table rows (suppress credential matches in pattern documentation) - Placeholder tokens (sk-..., ghp_xxxx, AKIA..., angle brackets) - Fail-safe: unclosed fences = unsuppressed (fail-open for security) Architecture: - Pure scanSkillContent() function exported for unit testing - Main flow wrapped in run() with import.meta.url guard - Existing checks unchanged; .copilot/skills skip for unsafe-git preserved Testing: 110 new tests covering all patterns, suppression, and golden corpus (zero false positives across 35 existing skill files). Closes #881, supersedes #879 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4076c6d to
af4993d
Compare
How This Works — Quick Guide for ReviewersWhat changed: How it runs: Same VM, same job, same step as the existing security review in What it catches (Phase 1 — high-confidence only):
What it does NOT catch (deferred to Phase 2):
Suppression (avoids false positives):
Enforcement level: Informational only. Posts a PR comment if findings exist. Does NOT block merge or fail CI. This is a heads-up, not a gate — appropriate for Phase 1 where suppression is simple. Test coverage: 110 new tests + golden corpus validation (zero false positives across all 35 existing skill files). Pure Cost: Zero additional Actions minutes. Zero new dependencies. ~3 seconds added to existing runner. PRD: #881 | Supersedes: #879 | Inspired by: microsoft/waza ComplianceChecker patterns |
tamirdresher
left a comment
There was a problem hiding this comment.
✅ Reviewed by Tamir + Squad.
Excellent security scanner — clean architecture with pure scanSkillContent() for testability. 30 patterns across 4 categories with 5-layer suppression is thorough. The fail-open on unclosed fences is the right security posture. 110 new tests + golden corpus validation (zero false positives on 35 existing skills) gives strong confidence.
Two minor Copilot notes (non-blocking):
- Unused statSync import in test file
- .squad/skills skip for unsafe-git already handled by the existing skip line
Phase 1 informational-only is the right approach. Ship it.
…882) Extends security-review.mjs with Tier 1 pattern detection for skill markdown files (.copilot/skills/**/*.md and .squad/skills/**/*.md): - P1: 10 embedded credential patterns (AWS, GitHub, OpenAI, JWT, etc.) - P2: 7 credential file read patterns (.env, .ssh/id_rsa, .aws/credentials, .npmrc, .netrc) - P3: 8 download-and-execute patterns (curl|bash, irm|iex, etc.) - P4: 5 privilege escalation patterns (sudo, RunAs, chmod 777, etc.) Suppression (Phase 1): - Fenced code blocks (CommonMark-compliant: backtick/tilde, variable length) - Inline code spans (backtick pairs) - Regex-doc table rows (suppress credential matches in pattern documentation) - Placeholder tokens (sk-..., ghp_xxxx, AKIA..., angle brackets) - Fail-safe: unclosed fences = unsuppressed (fail-open for security) Architecture: - Pure scanSkillContent() function exported for unit testing - Main flow wrapped in run() with import.meta.url guard - Existing checks unchanged; .copilot/skills skip for unsafe-git preserved Testing: 110 new tests covering all patterns, suppression, and golden corpus (zero false positives across 35 existing skill files). Closes #881, supersedes #879 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion security-review-skills.test.ts was added to main by PR #882 but references scanSkillContent which was subsequently removed. This test fails on main too. Removing from promotion branch since it's not part of insider. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Extends
security-review.mjswith Tier 1 pattern detection for skill markdown files (.copilot/skills/**/*.mdand.squad/skills/**/*.md). Zero new dependencies, zero workflow changes, ~2-3s added scan time.What's new
30 security patterns across 4 categories:
Phase 1 Suppression (5 layers):
Architecture:
scanSkillContent(content, filePath)function exported for testingrun()withimport.meta.urlguard (importing for tests doesn't trigger git/console side effects).copilot/skills/skip for unsafe-git (check Feature: Issue Assignment & Auto-Triage for Squad Members #4) preservedskill-credentials,skill-credential-file-read,skill-download-exec,skill-privilege-escalationTesting
110 new tests + 37 existing tests = 147 tests, all passing:
Not in scope (Phase 2)
Closes #881, supersedes #879
Working as Booster (CI/CD Engineer)