Skip to content

feat(cli): check-review findingFrames saves one plain frame per finding time - #6401

Merged
xuanruli merged 6 commits into
mainfrom
feat/check-finding-frames-v2
Oct 26, 2026
Merged

xuanruli merged 6 commits into
mainfrom
feat/check-finding-frames-v2

Conversation

@xuanruli

@xuanruli xuanruli commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What

A second check-review field in HF_TOOL_CONTEXT, findingFrames. When it is on, check saves one plain,
unannotated full frame for each distinct finding time, and adds two fields to each finding it covers:

  • frameTime: the time the frame was taken.
  • frame: the absolute path of that PNG. Findings at the same frameTime share one file.

Which findings get a frame: error and warning findings in the layout and contrast sections that have a real
bbox. motion and craft findings, info findings and findings without a bbox get no frame and no new fields.

Without the capability, output is unchanged.

Based on main. #6400 merged as 257bf285f on 2026-10-26 at 07:58:22 UTC. Landed main was merged normally into this branch in signed commit 0aacda324; the diff is only findingFrames.

Why

An agent reviewing check output gets a bbox and a time, but has to run snapshot itself to see what the finding
looks like, often at a time that does not show the problem. The existing annotated snapshots draw over the frame
and are taken at fixed sample times. A plain frame at a time when the finding is actually present lets a harness
crop the bbox or show the whole frame to a model without another command.

Related work

Builds on merged #6400, which adds the check-review capability. Builds on #6396.

How

  • Frame time: the middle sample of the longest contiguous run in the finding's own times, where two samples
    more than 1000 ms apart start a new run. This is the same gap rule heldMs uses, so the frame comes from the
    stretch where the finding persists longest rather than from a one-sample blip (checkFindingFrames.ts).
  • Capture: after the report is built, one browser session does one seek and one screenshot per distinct
    frameTime. It reuses the post-report capture path that --snapshots finding crops already use
    (withFindingCapturePage in checkBrowser.ts, now shared by crops and frames). When nothing qualifies, no
    session is launched.
  • Invalid samples: only finite, nonnegative times qualify; an empty eligible time list produces no frame. Cancellation during capture marks the check unsuccessful.
  • Cap: at most 60 distinct times. Times needed by error findings are kept first, then warnings. Findings whose
    time falls past the cap get no frame.
  • Storage: PNGs go in the run's snapshot directory. They are not added to snapshots, not recorded in project
    history, and not reported as artifact events.
  • Docs: the check-review table in docs/developers/harness-tool-context.mdx gets a findingFrames row.

Latency, wall clock of check --json, frames on vs this branch's base, in seconds:

project base frames
instagram_reel 5.55 6.70
hyperframe_update_highlights 4.30 5.11
google_playground_launch_film 4.12 4.12 (no qualifying finding, no capture)
bench_m1_dsh_sonnet 5.44 6.77

That is +0.8 to 1.3 s when any finding qualifies. Most of it is the second browser launch (0.6 to 0.9 s); the
seeks and screenshots are the rest. A run with no qualifying finding costs nothing.

Test plan

  • Unit tests added: check.test.ts covers selectFindingFrames (layout and contrast errors and warnings with
    a real bbox only, one frame per time; the cap keeps errors before warnings), middleOfLongestRun, turning
    frames on only through HF_TOOL_CONTEXT, and adding frame/frameTime only when on without touching
    snapshots. toolContext.test.ts covers parsing findingFrames.
  • vitest run src/commands/check.test.ts src/utils/toolContext.test.ts src/utils/layoutAudit.test.ts src/utils/checkBrowser.test.ts src/commands/checkArtifacts.test.ts
    in packages/cli: 7 files, 259 tests passed after main integration at 0aacda324; includes src/utils/checkSnapshotDirectory.test.ts and src/utils/checkDeadline.test.ts. New regressions for invalid time selection and cancellation both failed before their fixes. Earlier-stop deduplication (snapshots on/off) and a progressing 160-second fake-clock batch also fail before their fixes and pass after. Capture tests verify writes and cleanup with mocked browser sessions, including a later screenshot failure.
  • tsc --noEmit in packages/cli: clean. oxlint and oxfmt --check on changed TypeScript: clean. Comment citation and density checks pass against main. Initial workspace build passed using existing dependencies.
  • Manual: the latency table above, check --json on four real projects with the capability on. It was measured
    on the pre-squash branch, whose patch is identical to this PR's first commit; the second commit only renames a
    flag in two doc comments.
  • Documentation updated.
  • Comments follow CONTRIBUTING.md "Comments".

Not covered

  • bbox is measured at the finding's anchor time, not at frameTime. For an element that moves between the
    two, the box can sit somewhere else on the frame. This is documented in the field description.
  • No frames for motion or craft findings.
  • Independent followthrough validation did not rerun real-browser visual/latency measurements, full CI, coverage, live models, UI/Electron or Mintlify validation. The manual measurements above are author-reported evidence.

The --skip-contrast-in-motion flag and HYPERFRAMES_CHECK_SKIP_CONTRAST_IN_MOTION
env give way to a check-review capability in HF_TOOL_CONTEXT, so only a harness
turns it on. The skill text and manifest go back to main.
…ng time

With HF_TOOL_CONTEXT check-review.findingFrames, every error/warning finding in
layout or contrast with a real bbox gets frameTime (the middle sample of its
longest held run, by the heldMs gap rule) and frame (the absolute path of an
unannotated full frame shared by every finding at that time). One seek and
screenshot per distinct time, capped at 60 times with errors first, into the
run's snapshot folder; not listed in snapshots, history or events. Without the
capability, output is unchanged.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 2 important · 0 nits

Selection/cap, the shared capture page's cleanup and the frame/frameTime metadata look correct. Two deadline edges in the frames phase are worth fixing before this goes to main; details inline.

Scope

Read the full diff at 9ad7914 against feat/check-review-capability, plus checkDeadline.ts, checkSnapshotDirectory.ts and studio-server snapshotRuns.ts (frame run retention: the newest 10 runs or 24 h, and unrecorded runs aren't protected, which matches the documented behavior). Ran the 5 focused files locally: 246 passed. tsc --noEmit is clean. No other reviews were on the PR. Unit-test CI doesn't run against the stacked base, so it needs to pass after the retarget.

Comment thread packages/cli/src/utils/checkPipeline.ts Resolved
Comment thread packages/cli/src/utils/checkBrowser.ts Resolved
heygengenesis[bot]
heygengenesis Bot previously approved these changes Oct 26, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — 2 resolved, 0 open, 0 new · 9ad7914 → 5e14d8b. 249 focused tests pass locally. The frame-batch regression fails when the per-frame enter is removed.

Base automatically changed from feat/check-review-capability to main October 26, 2026 07:58
@xuanruli
xuanruli dismissed heygengenesis[bot]’s stale review October 26, 2026 07:58

The base branch was changed.

@mintlify

mintlify Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Oct 26, 2026, 8:09 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@xuanruli

Copy link
Copy Markdown
Contributor Author

Follow-up stacked on this branch: #6409 adds check-review.inspect. With findingFrames on, it attaches the component inspector's report to each framed finding.

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 144 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

  • seqplainkeys-none-px-r0-nested-z100: tracking 0.07, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.07, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.07, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE: 0 open, 0 new · 5e14d8b → 0aacda3 (main integration). The diff against main matches the patch approved at 5e14d8b, except for the checkReview() doc comment that #6400 removed on main. Main changed no other file this PR touches since the old base. 249 focused tests pass locally and the CLI typecheck is clean. Both CI runs and Windows render verification pass on this head.

@xuanruli
xuanruli added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit 12ae82a Oct 26, 2026
117 checks passed
@xuanruli
xuanruli deleted the feat/check-finding-frames-v2 branch October 26, 2026 08:45

This branch was successfully deployed

1 active deployment
staging - docs — 0aacda32 Deployed Oct 26, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant