Skip to content

feat: Adopt waza ComplianceChecker for skill security scanning #879

Closed

Description

@diberry

Summary

Replace the current band-aid in scripts/security-review.mjs (which skips .copilot/skills/**/*.md) with a proper markdown-aware security scanner inspired by microsoft/waza's ComplianceChecker framework.

Problem

The security review scanner (security-review.mjs) parses all files as JavaScript to detect unsafe patterns (git add ., credential exposure, etc.). When it encounters .md files containing literal git commands as documentation (e.g., SKILL.md files that explain git workflows), it throws SyntaxError because markdown isn't JavaScript.

Current band-aid: Skip .copilot/skills/**/*.md entirely (PR #873). This means skills get zero security scanning — not ideal since skills are executable instructions that agents follow.

Desired Behavior

Skills SHOULD be security-reviewed, but the scanner needs to understand markdown vs code:

File Type How to Scan
.js, .ts, .mjs Parse as JavaScript (current behavior)
.md (skills, instructions) Scan as text content — look for patterns in prose, not syntax
.yml, .yaml Scan for script injection patterns

Waza Patterns to Adopt

microsoft/waza implements exactly this with their ComplianceChecker interface:

1. OverSpecificityChecker (internal/checks/advisory_checks.go)

  • Flags hardcoded paths (/usr/, C:\), IPs, URLs, port numbers in skill body
  • Returns warnings, not parse errors
  • Squad equivalent: Catch credentials, hardcoded secrets, unsafe URLs in SKILL.md

2. SpecSecurityChecker (internal/checks/spec_checks.go)

  • Validates frontmatter for XML injection (angle brackets in values)
  • Checks reserved name prefixes
  • Squad equivalent: Validate skill metadata/frontmatter

3. NegativeDeltaRiskChecker

  • Flags conflicting instructions, excessive constraints in skill body
  • Squad equivalent: Catch skills that instruct agents to do unsafe things (git add ., git push --force, expose secrets)

4. Body vs Spec separation

  • Different checkers for markdown body vs YAML frontmatter
  • Squad equivalent: Don't parse .md as JavaScript — treat as text

Implementation Approach

  1. Add file-type detection to security-review.mjs
  2. For .md files: scan content as text using regex patterns (not AST parsing)
  3. Key patterns to detect in markdown:
    • git add . or git add -A recommended (not just mentioned as anti-pattern)
    • Hardcoded credentials, API keys, connection strings
    • --force push recommendations
    • Unsafe template expressions in workflow files
    • URLs pointing to internal/private endpoints
  4. Distinguish "warning against X" from "instruction to do X" (context-aware scanning)
  5. Remove the .copilot/skills/**/*.md skip once proper scanning is in place

Acceptance Criteria

  • .md files in .copilot/skills/ are scanned (skip removed)
  • Scanner doesn't throw SyntaxError on markdown files
  • Unsafe patterns in skill prose are detected and reported
  • Safe documentation about unsafe patterns (e.g., "never do X") is NOT flagged
  • Existing JS/TS scanning unchanged
  • Tests cover all file types

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or improvement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions