Skip to content

ci: gate on the benchmark, and say so when a run proves nothing - #13

Merged
tiltom merged 1 commit into
mainfrom
ci/benchmark-gates
Nov 19, 2026
Merged

tiltom merged 1 commit into
mainfrom
ci/benchmark-gates

Conversation

@tiltom

@tiltom tiltom commented Nov 19, 2026

Copy link
Copy Markdown
Member

scanner-core 0.7.0 published verdicts of HIGH (unsafe to clone) on
evanw/esbuild
and MEDIUM on reduxjs/redux. Nothing caught it: ci.yml runs
the unit tests, which were green, and the only record of what the corpus should
produce — benchmark/results/post-fix-final.json — was from 2026-10-21, a
month before the release, so a run had nothing to be compared against.

The benchmark could not have been a gate

npm run benchmark set process.exitCode only when something threw. A red
gate printed and exited 0 — so adding it to CI as-is would have produced a job
that passed no matter what the gates said.

--enforce-gates fails the run, on two things rather than one:

  • any gate false, or null on a full run — unevaluated is a skipped
    check, not a passed one. A --corpus / --sample run is exempt, since it
    cannot evaluate the gates belonging to corpora it left out.
  • a run that did not finish. This half is easy to overlook and it matters:
    a run that dies partway — usually an exhausted GitHub quota, which run.mjs
    breaks out of rather than throwing — leaves every sample it never reached
    out of the metrics. One such run was observed during this work reporting
    four green gates without ever reaching esbuild. That report is now a
    test fixture.

Samples missing their expected verdict are a warning, not a failure. The
five gates are the contract this benchmark declares; per-sample verdicts are a
finer signal, and failing on them would have CI refuse the very release that
fixes esbuild, because two Docusaurus false alarms remain. The corpus already
has a process for accepted false alarms (reference-review.json).

Three places now run it

run corpora token effect
ci.yml → benchmark-fixtures, every PR 3 in-memory fixture corpora none blocks the PR
benchmark.yml, nightly + manual all 114 FLAGRIX_GITHUB_TOKEN failed run
release.yml, before npm publish all 114 FLAGRIX_GITHUB_TOKEN blocks the publish

The fixture corpora are inert in-memory samples behind a mocked GitHub API — no
token, no network, seconds to run — but they cannot catch a false positive
that only shows on a real project
, which is exactly what 0.7.0's was. Hence
the public corpus gating the release too.

Nightly is not redundant with that: detection rules ship from
flagrix-detection-rules on their own cadence and reach the scanner with no
commit here
, so a rule can introduce a false positive that no pull request
touches and no release gate sees for months.

Action needed before the next release

Add a FLAGRIX_GITHUB_TOKEN secret (fine-grained, read-only PAT). Both
full-corpus runs check for it and fail with an explicit message if it is
missing. I deliberately did not fall back to the built-in GITHUB_TOKEN: it is
capped at 1,000 API requests per hour per repository, which a 114-sample run can
exhaust, and an incomplete run now fails the gate — so the fallback would turn a
missing secret into a confusing benchmark failure instead of a clear one.

Tests

The decision lives in benchmark/gates.mjs, apart from run.mjs, which
performs a run on import. tests/benchmark-gates.test.ts drives it with three
reports recorded during this episode — the 0.7.0 regression, the 0.7.1 run that
fixes it, and the partial run above — so the gate is pinned against the real
thing rather than a hand-written shape. 64 tests pass, typecheck clean, and
the three fixture corpora were run exactly as the new CI job runs them.

Deliberately not in here

post-fix-final.* is still the October snapshot. Regenerating it from a
symlinked working tree would contradict run.mjs's own rule — "metadata must
describe that artifact, not a sibling checkout that may be dirty or on a
different commit"
— so it waits for scanner-core 0.7.1 to publish, after which
it is npm install + a run + promote. The README now says to refresh it with
every engine release, and why a stale one is not a harmless stale file.

The offline mirror no longer matches an API-backed run. mirror.py keeps
its own copy of scanner-core's config patterns to choose which blobs to fetch,
and that copy predates 0.7.0 — no Dockerfile/Containerfile, compose,
.devcontainer, .claude/commands|agents|skills or .github/instructions. So
an offline run silently scans less than a real one. That is why CI uses the
fixture corpora and the API rather than the mirror, and the README now carries
the caveat. Bringing that list back in step is its own piece of work.

scanner-core 0.7.0 published verdicts of HIGH (unsafe to clone) on
evanw/esbuild and MEDIUM on reduxjs/redux. Nothing caught it: ci.yml runs the
unit tests, which were green, and the only record of what the corpus should
produce — benchmark/results/post-fix-final.json — was from 2026-10-21, a month
before the release, so a run had nothing to be compared against.

The benchmark could not have been wired in as it was. `npm run benchmark` set
process.exitCode only when something threw, so a red gate printed and exited
0. --enforce-gates makes the run fail, and it fails on two things, not one:

  - any gate false, or null on a full run (unevaluated is skipped, not passed;
    a --corpus or --sample run is exempt, since it cannot evaluate the gates
    belonging to corpora it left out)
  - a run that did not finish. This half is easy to lose and it matters: a run
    that dies partway, usually on an exhausted GitHub quota, which run.mjs
    breaks out of rather than throwing, leaves every sample it never reached
    out of the metrics. One such run was observed here reporting four green
    gates without ever reaching esbuild.

Samples missing their expected verdict are reported as a warning instead. The
five gates are the contract this benchmark declares; per-sample verdicts are a
finer signal, and failing on them would have CI refuse the very release that
fixes esbuild, because two Docusaurus false alarms remain. The corpus already
has a process for accepted false alarms (reference-review.json).

Three places now run it:

  - ci.yml, every PR: the three in-memory fixture corpora. No token, no
    network, fast. They cannot catch a false positive that only shows on a
    real project, which is what 0.7.0's was.
  - benchmark.yml, nightly and on demand: all 114 samples. Not only a backstop
    for the release gate — detection rules ship from flagrix-detection-rules
    on their own cadence and reach the scanner with no commit here, so a rule
    can introduce a false positive that no pull request touches.
  - release.yml, before npm publish: all 114 samples. The copy that can stop a
    bad version reaching the registry.

Both full-corpus runs need a FLAGRIX_GITHUB_TOKEN secret (fine-grained,
read-only PAT) and say so explicitly if it is missing. The built-in
GITHUB_TOKEN is capped at 1,000 API requests per hour per repository, which a
114-sample run can exhaust — and that now fails the gate, so falling back to
it would turn a missing secret into a confusing benchmark failure.

The decision itself is in benchmark/gates.mjs, apart from run.mjs, which
performs a run on import. tests/benchmark-gates.test.ts drives it with three
reports recorded during this episode: the 0.7.0 regression, the 0.7.1 run that
fixes it, and the partial run described above.

Not done here: post-fix-final.* is still the October snapshot. Regenerating it
from a symlinked working tree would contradict run.mjs's own rule that the
metadata must describe the artifact npm install resolved, so it waits for
scanner-core 0.7.1 to publish. README now says to refresh it with every engine
release, and why a stale one is not a harmless stale file.

Also noted in the README: the offline mirror no longer matches an API-backed
run. mirror.py keeps its own copy of scanner-core's config patterns to choose
which blobs to fetch, and that copy predates 0.7.0 — no Dockerfile, compose,
.devcontainer, .claude/commands|agents|skills or .github/instructions — so an
offline run silently scans less than a real one. That is why CI uses the
fixture corpora and the API rather than the mirror.
@tiltom
tiltom merged commit d13906a into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant