Skip to content

fix: --json carries the coverage verdict on incomplete scans - #7

Merged
tiltom merged 3 commits into
mainfrom
fix/json-coverage-verdict
Nov 9, 2026
Merged

tiltom merged 3 commits into
mainfrom
fix/json-coverage-verdict

Conversation

@tiltom

@tiltom tiltom commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Problem

flagrix scan --json printed scanner-core's raw result. When a scan ran out of GitHub budget over a low result, the JSON still said "riskLevel": "low" and "safeToClone": true, while the exit code was 4 and the human output said "SCAN INCOMPLETE — no verdict". An agent or script reading the JSON instead of the exit code would pass a repository that was barely read.

Change

scanReport() in src/lib/coverage.ts wraps the engine result for both --json and the scan_github_repo MCP tool's JSON block:

  • verdict: "low" | "medium" | "high" | "incomplete". It matches exit 0 / 2 / 3 / 4.
  • coverage: { complete, unreliableClear, filesRead, filesUnread, reason? }, the same object coverageOf() already computed for the exit code and the human output.
  • safeToClone is forced to false when coverage.unreliableClear is true.

All existing engine fields are kept, so consumers don't break. riskLevel still holds the engine's reading of the files it did see, so it can still say "low" on an incomplete scan. The docs now say to gate on verdict or the exit code instead.

Behaviour change for consumers: safeToClone used to be true on an incomplete low scan and is now false. It is unchanged in every other case.

scan-user needs no change. scanner-core's user scanner throws on any failed GitHub call (exit 1), so it never returns a partial result.

Wording: no "safe" claims

A scanner reduces risk; it can't certify a repository. The second commit changes the wording to match:

  • A complete low result now reads LOW RISK — No blocking findings instead of Safe to clone, which is what the MCP tool already said.
  • Exit 4 is now explained as "not enough of the repository was read to give a verdict" instead of "too little was read to call it safe", which implied that a complete scan would call it safe. This covers --help, the human output, the MCP guidance (now "only part of this repository was read"), the clone-gate hook, the README and docs/agent-gating.md.
  • The safeToClone JSON field name is unchanged, because scanner-core sets it and the browser extension reads it.

Docs: README "Incomplete scans (exit 4)", docs/agent-gating.md (the exit-code table now includes code 4, plus a note on gating on the JSON) and the --help text.

Tests

New JSON verdict and coverage block in tests/scan.test.ts:

  • complete low: exit 0, verdict: "low", safeToClone: true, full coverage
  • incomplete low (rate limit): exit 4, verdict: "incomplete", safeToClone: false, riskLevel still "low"
  • incomplete low (fetch errors): exit 4, reason: "fetch-errors"
  • incomplete medium: exit 2, verdict: "medium", coverage.complete: false, unreliableClear: false
  • MCP JSON block built from a raw engine result matches flagrix scan --json
  • human output for a complete low says "No blocking findings"; neither the human output nor the MCP text for a complete or incomplete scan contains "safe"

npm test (37/37), npm run typecheck and npm run build all pass.

Release

Version is 0.5.0 in package.json, package-lock.json, flagrix --version, the MCP server's reported version (src/commands/mcp.ts) and the MCP registry manifest (server.json, previously still at 0.1.0). It's a minor bump because the JSON output, the safeToClone behaviour on incomplete scans and the verdict wording all change. This PR doesn't publish to npm or the MCP registry.

`flagrix scan --json` printed scanner-core's raw result, so a scan that ran
out of GitHub budget over a low result said "riskLevel": "low" and
"safeToClone": true while the exit code was 4 and the human output said
"SCAN INCOMPLETE — no verdict".

The JSON (and the MCP tool's JSON block) now adds:
- `verdict`: "low" | "medium" | "high" | "incomplete", matching exit 0/2/3/4
- `coverage`: { complete, unreliableClear, filesRead, filesUnread, reason? }

and forces `safeToClone` to false when the low result is unreliable. All
existing engine fields are kept, so consumers don't break; `riskLevel` still
reports the engine's reading of the files it did see.

scan-user needs no change: a profile scan fetches everything or fails with
exit 1, so it never returns a partial result.
@tiltom tiltom self-assigned this Nov 9, 2026
A scanner reduces risk; it can't certify a repository, so the CLI stops
saying it can:

- A complete low result reads "LOW RISK — No blocking findings" instead of
  "Safe to clone", matching what the MCP tool already said.
- Exit 4 is explained as "not enough of the repository was read to give a
  verdict" instead of "too little was read to call it safe", which implied
  that a complete scan would. Changed in --help, the human output, the MCP
  guidance, the clone-gate hook, the README and docs/agent-gating.md.

The `safeToClone` JSON field is unchanged: scanner-core sets it and the
browser extension reads it.
Bumps the package, the lockfile, `flagrix --version`, the MCP server's
reported version and the MCP registry manifest (server.json, which was
still at 0.1.0) to 0.5.0.

Minor rather than patch: `--json` gains `verdict` and `coverage`,
`safeToClone` turns false on incomplete scans, and the human-readable
verdict wording changes.
@tiltom
tiltom merged commit f70e9ce into main Nov 9, 2026
3 checks passed
@tiltom
tiltom deleted the fix/json-coverage-verdict branch November 18, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant