[mvanhorn]

Rebased version of #3756, which was closed as stale (it couldn't be reopened). Rebased on current main. The only conflict was docs/docs.json nav entries, plus one test fixture fix for the new browserSkipped and hdr fields on CheckReport. Heads up: the committed example baselines came from Linux, and on macOS the text antialiasing alone trips the gate. Happy to loosen the example's tolerance or document per-platform baselines.

What

A committed-baseline visual regression gate for compositions:

• Convention: reference frames live at golden/<compositionId>/<timeMs>.png with an optional golden/<compositionId>/golden.json manifest (times, threshold, maxDiffRatio, ignoreAntialiasing).
• hyperframes check --golden re-captures the composition at the manifest times, pixel-diffs each frame against its baseline (configurable per-channel threshold, 1px anti-aliasing shift detection), fails the check on regression, and writes golden-diff/ with per-frame red-highlight diff PNGs plus a baseline | current | diff contact sheet.
• hyperframes check --update-golden / hyperframes snapshot --update-golden refresh the baselines and manifest.
• Agent-readable JSON summary embedded in the check --json envelope: {ok, failed:[{id, time, timeMs, maxDelta, diffRatio, reason}], diffSheet, ...}.
• Runnable example with committed baselines under examples/golden-baseline/; docs at docs/reference/cli-golden.mdx and docs/guides/visual-regression.mdx.

Why

Agents regress layouts silently: a refactor nudges a headline, a token change recolors a button — and every command still exits 0. HyperFrames already has snapshot and compare, but no committed golden baseline and no fail-on-diff CI gate. This turns "the video still looks right" into a hard, committable contract with one exit code.

How

• packages/cli/src/golden/pixelDiff.ts — pure diff math on raw RGBA buffers via sharp (already a CLI dependency; no new dependencies): per-channel threshold, neighborhood-based anti-aliasing classification, red/amber/pale-gray diff visualization.
• packages/cli/src/golden/baseline.ts — the golden/ convention, manifest parsing/validation, sample-time resolution (--at → manifest → baseline filenames → default spread), and the gate/update orchestration. Capture reuses the same font-localized, settled-page path as snapshot/compare (openSettledCompositionPage + seekCompositionTimeline).
• packages/cli/src/capture/contactSheet.ts — new createGoldenDiffContactSheet (one baseline | current | diff row per failed time).
• check/snapshot wiring: --golden, --update-golden, --golden-threshold; golden failures combine with the normal check exit code; a missing baseline fails with instructions rather than silently passing.
• Dimension changes always fail regardless of the pixel budget; golden-diff/ is derived output and git-ignored in the example.

Walkthrough (HyperFrames-rendered)

The walkthrough is itself a HyperFrames composition rendered to MP4 with this branch's CLI — storyboarded first, authored as HTML, validated with hyperframes lint + hyperframes check (0 errors, 33/33 WCAG AA text checks), then rendered with:

npx hyperframes render . --quality high --no-browser-gpu -o renders/golden-gate-walkthrough.mp4
# ◇ 2.8 MB · 48.0s video · rendered in 50.5s (1440 frames @ 30fps)

Every terminal line and every frame image in the film is real CLI output from examples/golden-baseline: the three committed golden PNGs, the failing run's current frame, its red-highlight diff, and the generated contact sheet.

▶ Watch the walkthrough (MP4, 48s, 2.8 MB)

Stills from the rendered film:

baselines frozen → gate passes	one hex of drift → gate fails	arrival: the diff sheet

The real diff contact sheet produced by the failing run (golden-diff/golden-baseline/contact-sheet.jpg):

Test plan

• Unit tests added/updated — packages/cli/src/golden/pixelDiff.test.ts (diff math on synthetic PNGs, no Chrome: thresholds, AA classification, dimension mismatch, PNG round trip), packages/cli/src/golden/baseline.test.ts (path conventions, manifest validation, sample-time precedence, gate verdicts), check.test.ts (flag grammar + command wiring/exit codes). Full @hyperframes/cli suite: 213 files, 3034 tests passing (bun run test).
• Manual testing performed — on examples/golden-baseline: created baselines (snapshot --update-golden), verified check --golden passes across independent captures, tweaked the accent color #f5a623 → #e0245e, verified the gate fails with exit 1 + diff sheet + JSON summary, then refreshed with check --update-golden.
• Documentation updated — docs/reference/cli-golden.mdx, docs/guides/visual-regression.mdx, registered in docs/docs.json; example README.
• bunx oxlint / oxfmt clean; tsc --noEmit clean; fallow audit clean (no new dead code, cycles, duplication, or complexity findings).

AI disclosure

This PR was authored by an AI agent (Claude Fable 5 via Cursor Cloud Agents), including the code, tests, docs, example, and the HyperFrames-rendered walkthrough video. It was developed and verified end-to-end as described in the test plan; per CONTRIBUTING, correctness responsibility is accepted by the submitter.

[mvanhorn]

Merged main to clear the conflict in check.ts, keeping main's version of the frame-check handling (swallowedFlagUsageError, no normalizeFrameCheckRawArgs) and carrying only the golden gate wiring over. I also cut the comments this PR added to check.ts in f5874ce, which is what the Comments check was flagging. Lint, typecheck, the comment ratchet and the check/golden tests are green on my end.

[miguel-heygen]

Thanks for bringing this back with a fuller write-up. Reviewed at f5874ce.

Where it stands: main has moved a lot under check since your merge. packages/cli/src/commands/check.ts now conflicts in five places, along with docs/docs.json. Since then check gained scoped checks, a project lock with per-revision artifacts, warm Chrome reuse and --prepare-media, so the golden wiring in executeCheck/emitCheckReport needs re-threading into the new flow rather than a plain re-merge. The green CI here is from before all of that. Nothing on main does a committed-baseline gate, so the idea isn't superseded.

Two things I'd settle before the rebase:

1. Where baselines come from. You already noted that the Linux-made example baselines fail on macOS from text antialiasing alone. The capture is a page.screenshot in the local snapshot browser, so a committed baseline only holds on the OS that made it, and a CI gate on such files fails for anyone on another platform. This is a design call for the maintainers. For example, baselines could only be created and checked inside the pinned render image, or stored per platform. It's worth agreeing on before more code depends on the golden/ layout.
2. snapshot --update-golden --at <t> replaces the whole baseline set. writeGoldenBaselines (packages/cli/src/golden/baseline.ts:355) removes any baseline whose time wasn't captured in this run and rewrites times to the captured set. So refreshing one frame with snapshot --update-golden --at 2 on the example removes 500.png and 3500.png and leaves times: [2]. The flag help does say --at overrides the sample times, but someone refreshing a single frame will probably expect the others to stay. Merging into the existing set, or at least printing which baselines were removed, would avoid the surprise.

Verdict: COMMENT
Reasoning: The branch conflicts with the reworked check command and the cross-platform baseline question needs a maintainer decision first; one update-path footgun is noted above.

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 03a9fb7.

Approve preview

[mvanhorn]

Thanks for the close read. 562f93b fixes the partial update and merges current main.

snapshot --update-golden --at <t> now merges into the existing set. writeGoldenBaselines writes the captured frames and adds their times to the manifest times and the existing baseline filenames instead of deleting the rest, so --at 2 on the example rewrites 2000.png and keeps 500.png, 3500.png and times: [0.5, 2, 3.5]. There's a test for it, and the flag help and the cli-golden reference now say so.

For the merge I kept main's check flow and put the golden gate on top of it: it runs inside the project lock after the pipeline, --prepare-media still exits before it, and the saved report and revision lines still print. snapshot --update-golden takes the project lock too, and the check flag table in docs/packages/cli.mdx lists the golden flags. On that commit, oxfmt --check, oxlint, fallow audit, the CLI typecheck and vitest over the golden and command tests all pass locally.

On where baselines come from, that's your call as maintainers. Which do you prefer: baselines created and checked only inside the pinned render image, or baselines stored per platform?

[miguel-heygen]

Thanks for the update. The partial refresh now keeps the other baselines, and the test covers it. The merge with the reworked check also looks right: --prepare-media still exits first, the gate runs inside the project lock after the pipeline, and the report and revision lines still print.

Two checks are red on this branch:

1. Build: check:cli-lazy-deps. packages/cli/src/golden/pixelDiff.ts:9 imports sharp statically, and the CLI only allows it as a lazy load. contactSheet.ts shows the pattern: const sharp = await loadSharp("contact sheets") from ../utils/optionalPackages.js.
2. Comments. pixelDiff.ts is a new file at 11.4% comment lines against the package's 11.0%. Cutting a line or two gets it under.

Two smaller things:

• The saved report doesn't include the golden result. runPipeline saves the report and revision before the golden gate runs, and the golden result only reaches the terminal and --json output. So check --golden with a regressed frame prints "Check failed" and exits 1, but the file printed after Report: says the check passed. Running the gate before the save, or adding golden to the report that gets saved, would keep them in step.
• Removing a time from golden.json no longer works on its own. A full --update-golden samples the manifest times, then the merge adds back every PNG still in the folder. If someone removes 3.5 from times and refreshes, 3.5 comes back and 3500.png keeps the old frame, so the next check --golden compares against a stale baseline. Merging only when --at is given, and replacing the set on a full refresh, would keep both cases working. Otherwise, documenting "delete the PNG to drop a time" would do.

On where baselines come from: thanks for laying out the two options. That decision is for the maintainers, and it's still the main thing to settle before this can land.

Verdict: COMMENT
Reasoning: The conflict and partial-update issues are fixed, but Build and Comments fail because of this branch, and the cross-platform baseline question still needs a maintainer decision.

[miguel-heygen]

Thanks, this round fixes everything I raised, and CI is green.

• sharp now loads lazily through loadSharp, and the comment share is under.
• The golden result is merged into the report before it's saved, so the saved file, the terminal, --json and the exit code all read the same ok. Passing it in as a finish step keeps the save in one place.
• A full --update-golden replaces the set again and only --at merges, with a test for dropping a time from the manifest and a docs update.

From the code side I have nothing further. What's left is the question you asked about where baselines come from (created and checked only in the pinned render image, or stored per platform). That's for the maintainers to answer, and it's the main thing to settle before this lands.

Verdict: COMMENT
Reasoning: All earlier code findings are fixed and CI is green; the baseline-source decision is still open for the maintainers.

[mvanhorn]

Both red checks were mine, and both smaller points are fixed in a0ed431.

1. Build: pixelDiff.ts now loads sharp with await loadSharp("golden baselines"), like contactSheet.ts. check:cli-lazy-deps passes on a local build.
2. Comments: I removed the header comment in pixelDiff.ts. CI's Comments check passes now.

• The saved report now includes the golden result. runPipeline takes an optional finish step that runs the golden gate inside the project lock, between the check pipeline and the save. So the saved report, the --json output and the terminal all get the same golden and the same combined ok. There's a test that the report handed to the save step has ok: false and the golden summary when a frame regresses.
• A full --update-golden replaces the set again. It writes the sampled frames, deletes PNGs for times it didn't sample, and sets times to what it captured. So removing 3.5 from golden.json and refreshing drops 3500.png. Only snapshot --update-golden --at merges into the existing set. There's a test for each case, and the --update-golden row in the cli-golden reference now says removed times lose their baselines.

On a0ed431, every CI check that ran passed, including Build, Comments, Typecheck, Lint and Format. Thanks for confirming the baseline question is with the maintainers.

[jrusso1020]

Requesting changes for one security issue: the composition id is used as a path segment without validation.

data-composition-id is read from the rendered page after the project's own scripts run (packages/cli/src/golden/baseline.ts:315), and used as-is (:320). It then becomes a path segment in goldenCompositionDir (:108, join(projectDir, "golden", compositionId)) and in the diff directory, which is deleted with rmSync(join(project.dir, "golden-diff", compositionId), { recursive: true, force: true }) (:405-406).

So a project whose composition sets data-composition-id="../.." (or a script that sets it at runtime) makes hyperframes check --golden recursively delete a directory outside the project. --update-golden writes PNGs and golden.json to the same escaped path, and deletes digit-named .png files there. The gate is opt-in, but people run check on projects they didn't write (templates, shared repos, agent output).

Fix: validate the id against an allowlist pattern before using it, the way frame ids already are (SAFE_FRAME_ID_PATTERN), and also assert the resolved directory stays inside project.dir before any rmSync or write. A test with data-composition-id="../x" covering both --golden and --update-golden would pin it.

Everything else in the security pass looked fine, including the three baseline PNGs (valid 640x360, no trailing data). The broader feature still needs a maintainer product call.

— Rames

[jrusso1020]

Direction is good. Once the path issue from the review is fixed we'll take it. For baselines, start with per-platform since that's what people will actually have locally.

[mvanhorn]

fb205b8 brings current main into the branch. The one conflict was in check.ts: main added recordLastCheck around the pipeline and the lint-aware browserSkippedLine, and the branch had moved that flow into executeCheck with the golden finish step. executeCheck now records the last check before and after the locked pipeline run, with the exit code and failure codes taken from the report that includes the golden result. printHumanReport keeps main's skipped-browser line and Audio section plus the golden section. The golden wiring test fixture also gained the audio section that CheckReport now requires.

On the merge, the CLI typecheck shows no new errors, oxfmt --check and oxlint are clean on the changed files, and vitest over the golden, check and snapshot tests gives 351 passed.

This push is only the merge. The composition id path fix from the security review is still to come in its own commit.

[mvanhorn]

The red CLI jobs came from one test, cli.agentMode.test.ts > check --range 5,1 exits 2 with a usage_error envelope. My refactor of executeCheck resolved the project before parsing flags, so a bad --range in a folder without a composition exited 1 with "No composition found" instead of the usage error. 8764045 parses --range and the golden flags first, the same order main uses.

That case fails before the change and passes after it, and check.test.ts passes 162/162 locally.

[mvanhorn]

Pushed 348084f. The composition id is now checked against an allowlist, and the golden/ and golden-diff/ folders must stay inside the project before any write or delete, for both --golden and --update-golden. Tests cover ../x and ../../x. Per-platform baselines are not in this push yet.

[mvanhorn]

@jrusso1020 for the per-platform baselines, would you prefer golden/<platform>/<compositionId>/ or golden/<compositionId>/<platform>/? The first would move the existing Linux example baselines under golden/linux/. I will build whichever you pick.

[mvanhorn]

0ac84ed merges current main. The only conflict was docs/docs.json: main moved the Developers pages under the Docs tab, so I kept main's layout and put guides/visual-regression and reference/cli-golden back in the same groups. There were no code conflicts.

On the merge, oxfmt --check, oxlint, bun run build and the typecheck across all packages pass, and the golden, check, snapshot and agent mode tests pass 774/774.

The full CLI suite was not clean locally. 28 tests failed in a run on a heavily loaded machine, and 19 of those passed when I reran the failing files one at a time. The other 9 are in the offload audio track, sherpa, whisper manager, project lock and checkSweep WebGL tests. The same 9 fail on a clean checkout of main on this machine, so they are not from this branch.

This push is only the merge. Per-platform baselines still depend on the layout question above.

[mvanhorn]

03a9fb7 merges current main. The conflicts were in check.ts and check.test.ts: main added the check pass cache (checkOrReplay), the report path notice and --include-unmounted, so executeCheck now runs through checkOrReplay and a --golden or --update-golden run skips the cache, since the cache key does not cover the baselines. On the merge, the build, CLI typecheck, oxfmt, oxlint and the check, golden, snapshot and agent mode tests pass locally.