Repository navigation
test(studio): edit accuracy screenshots are viewport-only and taken in the confirmation re-runs - #6795
Merged
Conversation
Contributor
Edit accuracy: accurate 271 (base branch 271), smooth 112 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (1)
|
Contributor
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 2 important · 0 nits · head 151160ab
Not a fix PR, so no 'fails without the fix' line is needed. CI is green at this head (edit accuracy gate and Fallow audit included).
- important, the body is out of date:
151160abalso stops first runs from taking evidence screenshots. Only--rerun(runs 2 and 3 of flipped cases) and--shotstake them now, and the error screen is always kept. The title, body and timing table describe only the viewport-only change. The tradeoff is also missing: a case that fails on base too (not flipped) no longer gets any screens in CI. Please add this to the body. AGENTS.md already documents it.
Scope
Read case.mjs, run.mjs, sampling.test.mjs and AGENTS.md in full against `scripts/ci/edit-accuracy-shard.sh` and the retry and gate jobs in ci.yml. The shots are written only as files (run.mjs:52), and no comparison reads them.
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Each edit accuracy case took up to three evidence screenshots (after the commit, after the undo, after the reload). Two changes:
captureBeyondViewportto true, so Chrome resized the page for every shot. Studio fills the viewport, so a viewport-only capture is the same image. Shots now passcaptureBeyondViewport: falseandoptimizeForSpeed: true.--rerun), so screenshots now come from those re-runs. An error screen is always kept.--shotsturns them on for a first run, for local debugging.What a reviewer gets for each kind of failure
results.json. If the render was wrong, the producer frame too.--shotslocally.results.jsonkeeps the case's frame intervals, main-thread work per frame, dropped frames and the blank-page control. Screens never carried these.slow-runner.jsonwith the first cases' times.Why
Probe. A temporary push-only workflow on a throwaway branch (since deleted) ran 24 cases with phase timers on 3 hosted runners. With full-page captures (run 41006449908), the commit and undo shots alone took a median 4.6 s per case (hold, zoom 100) to 8.7 s (no GSAP, zoom 50). Viewport-only (run 41007529098) cut whole cases from 19.2 to 15.6 s, 23.7 to 17.7 s, 22.4 to 16.2 s and 30.6 to 25.0 s across the four groups. Phases the change cannot touch stayed within 10% between those runs. With full-page shots out, finding the composition after the reload got up to 12x faster: the resize had been disturbing Studio mid-measurement.
This PR's own run (41008932028) against #6784's run (41005139661); both use 15 packed shards on the 271-case PR slice:
One run cannot separate runner speed from the change. The drop matches the costs the probe measured phase by phase.
No visible change
Only the edit accuracy bench under
packages/studio/tests/andpackages/studio/AGENTS.mdchange.Checks
bunx vitest run packages/studio/tests/e2e/edit-accuracy/sampling.test.mjs packages/studio/tests/e2e/edit-accuracy/settle.test.mjs: 18/18, three runs in a row.skips a first run's screenshots but keeps its error screenfails when the first-run skip is removed.takes a confirmation run's screenshots, of the viewport onlypinscaptureBeyondViewport: false.fallow auditpass. The edit accuracy gate on this PR is green.Size
Small on purpose: one change to when edit-accuracy cases take evidence screenshots, measured on real CI runs. It stands alone from the other CI speedups so each can be reverted on its own.