Repository navigation
ci: gate on the benchmark, and say so when a run proves nothing - #13
Merged
Merged
Conversation
scanner-core 0.7.0 published verdicts of HIGH (unsafe to clone) on
evanw/esbuild and MEDIUM on reduxjs/redux. Nothing caught it: ci.yml runs the
unit tests, which were green, and the only record of what the corpus should
produce — benchmark/results/post-fix-final.json — was from 2026-10-21, a month
before the release, so a run had nothing to be compared against.
The benchmark could not have been wired in as it was. `npm run benchmark` set
process.exitCode only when something threw, so a red gate printed and exited
0. --enforce-gates makes the run fail, and it fails on two things, not one:
- any gate false, or null on a full run (unevaluated is skipped, not passed;
a --corpus or --sample run is exempt, since it cannot evaluate the gates
belonging to corpora it left out)
- a run that did not finish. This half is easy to lose and it matters: a run
that dies partway, usually on an exhausted GitHub quota, which run.mjs
breaks out of rather than throwing, leaves every sample it never reached
out of the metrics. One such run was observed here reporting four green
gates without ever reaching esbuild.
Samples missing their expected verdict are reported as a warning instead. The
five gates are the contract this benchmark declares; per-sample verdicts are a
finer signal, and failing on them would have CI refuse the very release that
fixes esbuild, because two Docusaurus false alarms remain. The corpus already
has a process for accepted false alarms (reference-review.json).
Three places now run it:
- ci.yml, every PR: the three in-memory fixture corpora. No token, no
network, fast. They cannot catch a false positive that only shows on a
real project, which is what 0.7.0's was.
- benchmark.yml, nightly and on demand: all 114 samples. Not only a backstop
for the release gate — detection rules ship from flagrix-detection-rules
on their own cadence and reach the scanner with no commit here, so a rule
can introduce a false positive that no pull request touches.
- release.yml, before npm publish: all 114 samples. The copy that can stop a
bad version reaching the registry.
Both full-corpus runs need a FLAGRIX_GITHUB_TOKEN secret (fine-grained,
read-only PAT) and say so explicitly if it is missing. The built-in
GITHUB_TOKEN is capped at 1,000 API requests per hour per repository, which a
114-sample run can exhaust — and that now fails the gate, so falling back to
it would turn a missing secret into a confusing benchmark failure.
The decision itself is in benchmark/gates.mjs, apart from run.mjs, which
performs a run on import. tests/benchmark-gates.test.ts drives it with three
reports recorded during this episode: the 0.7.0 regression, the 0.7.1 run that
fixes it, and the partial run described above.
Not done here: post-fix-final.* is still the October snapshot. Regenerating it
from a symlinked working tree would contradict run.mjs's own rule that the
metadata must describe the artifact npm install resolved, so it waits for
scanner-core 0.7.1 to publish. README now says to refresh it with every engine
release, and why a stale one is not a harmless stale file.
Also noted in the README: the offline mirror no longer matches an API-backed
run. mirror.py keeps its own copy of scanner-core's config patterns to choose
which blobs to fetch, and that copy predates 0.7.0 — no Dockerfile, compose,
.devcontainer, .claude/commands|agents|skills or .github/instructions — so an
offline run silently scans less than a real one. That is why CI uses the
fixture corpora and the API rather than the mirror.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scanner-core 0.7.0 published verdicts of HIGH (unsafe to clone) on
evanw/esbuild and MEDIUM on reduxjs/redux. Nothing caught it:
ci.ymlrunsthe unit tests, which were green, and the only record of what the corpus should
produce —
benchmark/results/post-fix-final.json— was from 2026-10-21, amonth before the release, so a run had nothing to be compared against.
The benchmark could not have been a gate
npm run benchmarksetprocess.exitCodeonly when something threw. A redgate printed and exited 0 — so adding it to CI as-is would have produced a job
that passed no matter what the gates said.
--enforce-gatesfails the run, on two things rather than one:false, ornullon a full run — unevaluated is a skippedcheck, not a passed one. A
--corpus/--samplerun is exempt, since itcannot evaluate the gates belonging to corpora it left out.
a run that dies partway — usually an exhausted GitHub quota, which
run.mjsbreaks out of rather than throwing — leaves every sample it never reachedout of the metrics. One such run was observed during this work reporting
four green gates without ever reaching esbuild. That report is now a
test fixture.
Samples missing their expected verdict are a warning, not a failure. The
five gates are the contract this benchmark declares; per-sample verdicts are a
finer signal, and failing on them would have CI refuse the very release that
fixes esbuild, because two Docusaurus false alarms remain. The corpus already
has a process for accepted false alarms (
reference-review.json).Three places now run it
ci.yml→benchmark-fixtures, every PRbenchmark.yml, nightly + manualFLAGRIX_GITHUB_TOKENrelease.yml, beforenpm publishFLAGRIX_GITHUB_TOKENThe fixture corpora are inert in-memory samples behind a mocked GitHub API — no
token, no network, seconds to run — but they cannot catch a false positive
that only shows on a real project, which is exactly what 0.7.0's was. Hence
the public corpus gating the release too.
Nightly is not redundant with that: detection rules ship from
flagrix-detection-rules on their own cadence and reach the scanner with no
commit here, so a rule can introduce a false positive that no pull request
touches and no release gate sees for months.
Action needed before the next release
Add a
FLAGRIX_GITHUB_TOKENsecret (fine-grained, read-only PAT). Bothfull-corpus runs check for it and fail with an explicit message if it is
missing. I deliberately did not fall back to the built-in
GITHUB_TOKEN: it iscapped at 1,000 API requests per hour per repository, which a 114-sample run can
exhaust, and an incomplete run now fails the gate — so the fallback would turn a
missing secret into a confusing benchmark failure instead of a clear one.
Tests
The decision lives in
benchmark/gates.mjs, apart fromrun.mjs, whichperforms a run on import.
tests/benchmark-gates.test.tsdrives it with threereports recorded during this episode — the 0.7.0 regression, the 0.7.1 run that
fixes it, and the partial run above — so the gate is pinned against the real
thing rather than a hand-written shape. 64 tests pass,
typecheckclean, andthe three fixture corpora were run exactly as the new CI job runs them.
Deliberately not in here
post-fix-final.*is still the October snapshot. Regenerating it from asymlinked working tree would contradict
run.mjs's own rule — "metadata mustdescribe that artifact, not a sibling checkout that may be dirty or on a
different commit" — so it waits for scanner-core 0.7.1 to publish, after which
it is
npm install+ a run + promote. The README now says to refresh it withevery engine release, and why a stale one is not a harmless stale file.
The offline mirror no longer matches an API-backed run.
mirror.pykeepsits own copy of scanner-core's config patterns to choose which blobs to fetch,
and that copy predates 0.7.0 — no Dockerfile/Containerfile, compose,
.devcontainer,.claude/commands|agents|skillsor.github/instructions. Soan offline run silently scans less than a real one. That is why CI uses the
fixture corpora and the API rather than the mirror, and the README now carries
the caveat. Bringing that list back in step is its own piece of work.