Skip to content

test(studio): edit accuracy screenshots are viewport-only and taken in the confirmation re-runs - #6795

Merged
miguel-heygen merged 3 commits into
mainfrom
test/edit-accuracy-viewport-shots
Oct 30, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
test/edit-accuracy-viewport-shots

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Each edit accuracy case took up to three evidence screenshots (after the commit, after the undo, after the reload). Two changes:

  1. Viewport only. Puppeteer 25 defaults captureBeyondViewport to true, so Chrome resized the page for every shot. Studio fills the viewport, so a viewport-only capture is the same image. Shots now pass captureBeyondViewport: false and optimizeForSpeed: true.
  2. Only in the confirmation re-runs. A first run keeps its screenshots only on failure, and every case the gate can fail on is re-run twice (--rerun), so screenshots now come from those re-runs. An error screen is always kept. --shots turns them on for a first run, for local debugging.

What a reviewer gets for each kind of failure

failure what the shard artifact holds
accuracy regression (fails 2 of its 3 runs; this is what fails the gate) From the re-runs: screens after commit, undo and reload, plus the saved files. From every run: the measured metrics in results.json. If the render was wrong, the producer frame too.
case already failing on main (no flip, never gated) Saved files and metrics; no screens. Use --shots locally.
smoothness only (reported, never gated) Every run's results.json keeps the case's frame intervals, main-thread work per frame, dropped frames and the blank-page control. Screens never carried these.
harness error The error screen, the error with its stack, and the end of the server log.
slow runner slow-runner.json with the first cases' times.

Why

Probe. A temporary push-only workflow on a throwaway branch (since deleted) ran 24 cases with phase timers on 3 hosted runners. With full-page captures (run 41006449908), the commit and undo shots alone took a median 4.6 s per case (hold, zoom 100) to 8.7 s (no GSAP, zoom 50). Viewport-only (run 41007529098) cut whole cases from 19.2 to 15.6 s, 23.7 to 17.7 s, 22.4 to 16.2 s and 30.6 to 25.0 s across the four groups. Phases the change cannot touch stayed within 10% between those runs. With full-page shots out, finding the composition after the reload got up to 12x faster: the resize had been disturbing Studio mid-measurement.

This PR's own run (41008932028) against #6784's run (41005139661); both use 15 packed shards on the 271-case PR slice:

before after
case time, median / mean 24.8 / 29.6 s 16.3 / 18.1 s
shard jobs 186 to 427 s 153 to 287 s
edit accuracy gate finished 650 s 492 s
first-run screenshots saved 504 1 (an error screen)

One run cannot separate runner speed from the change. The drop matches the costs the probe measured phase by phase.

No visible change

Only the edit accuracy bench under packages/studio/tests/ and packages/studio/AGENTS.md change.

Checks

  • bunx vitest run packages/studio/tests/e2e/edit-accuracy/sampling.test.mjs packages/studio/tests/e2e/edit-accuracy/settle.test.mjs: 18/18, three runs in a row.
    • skips a first run's screenshots but keeps its error screen fails when the first-run skip is removed.
    • takes a confirmation run's screenshots, of the viewport only pins captureBeyondViewport: false.
  • oxlint, oxfmt, comment citations and fallow audit pass. The edit accuracy gate on this PR is green.

Size

Small on purpose: one change to when edit-accuracy cases take evidence screenshots, measured on real CI runs. It stands alone from the other CI speedups so each can be reverted on its own.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 112 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-center-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 2 important · 0 nits · head 151160ab

Not a fix PR, so no 'fails without the fix' line is needed. CI is green at this head (edit accuracy gate and Fallow audit included).

  • important, the body is out of date: 151160ab also stops first runs from taking evidence screenshots. Only --rerun (runs 2 and 3 of flipped cases) and --shots take them now, and the error screen is always kept. The title, body and timing table describe only the viewport-only change. The tradeoff is also missing: a case that fails on base too (not flipped) no longer gets any screens in CI. Please add this to the body. AGENTS.md already documents it.
ScopeRead case.mjs, run.mjs, sampling.test.mjs and AGENTS.md in full against `scripts/ci/edit-accuracy-shard.sh` and the retry and gate jobs in ci.yml. The shots are written only as files (run.mjs:52), and no comparison reads them.

@miguel-heygen miguel-heygen changed the title test(studio): edit accuracy evidence screenshots capture only the viewport test(studio): edit accuracy screenshots are viewport-only and taken in the confirmation re-runs Oct 30, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE stands: 1 resolved (body), 1 open (case.mjs:992, inStudio JSDoc + fallow-ignore orphaned above shooter, code, not body), 0 new · 151160ab → 151160ab

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE: 2 resolved, 0 open, 0 new · 151160ab → 551e6b70. All checks green.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 36cf990 Oct 30, 2026
128 checks passed
@miguel-heygen
miguel-heygen deleted the test/edit-accuracy-viewport-shots branch October 30, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant