Repository navigation
test(engine): pixel invariants and fixtures that guard default-on capture blur - #7108
Conversation
There was a problem hiding this comment.
Verdict: REQUEST CHANGES · 1 blocker · 1 important · 1 nit · head ab76d6d
The first Linux run is red in both lanes, and every failure is held DOM text on a capture-tier fixture. Details are inline on thresholds.ts. The other two are timing and dead-field issues.
CI evidence and durations (run 41711675131)
- Test (runtime): 9 of 73 differential cases fail:
callbacks,stateful-callbackandoverflow-wordmarkoutside cases at every checkpoint. Worst tiles have MAE 1.8–10 on the text rows. The pixels (18), fixtures (29) and liveness (15) files pass, andsupportedwas true, so the suite really ran. Job time is 8m08s against a 10-minute cap; recent main runs take about 6.5–7 min. The differential file adds 48.6 s, liveness 13.5 s and fixtures 8.4 s. - Producer: integration tests: 4m38s against a 20-minute cap. Three blur cases fail with the same signature:
overflow-wordmarkheld frames 13–17 (MAE 2.2–3.2),callbacksframes 18–22 on the label (4.2–6.0) andscroll-listframes 2–6 in the list (2.2–7.2). Theblur off twicenoise cases pass, so this is not render nondeterminism. The 4th failure (coreRuntimeBrowser.test.tsmemory, 4695 vs 800) is in a file this PR doesn't touch; I haven't checked whether it's flaky on main. - Shutter centring in
shutterTimesmatchesframes.ts:125(t - shutter/2).
Audited: all 11 files.
— Genesis
Edit accuracy: accurate 267 (base branch 267), smooth 109 of thoseThe gate passes. Quarantined, measured but not gated (8)
Unstable (2)
|
6a36ed9 to
1d1ab8b
Compare
|
Linux CI results for this PR. Three green runs of both lanes (Chrome 153 pin, ubuntu-latest):
What the first Linux run found, all fixed on this branch:
Not from this PR: Not covered yet, as stated in the description: the shutter-angle and weak-blur cases, performance, Windows rendering, and Rames's 60 compositions. |
|
All three addressed (head
Also pushed: the blur helpers are out of the engine tarball (size budget), and the reachability runner hash is re-pinned. |
| }); | ||
| } | ||
|
|
||
| describe.skipIf(!HAS_FFMPEG)("page motion blur through a multi-worker render", () => { |
There was a problem hiding this comment.
important: Flake gate hits its 30-minute job limit and gets cancelled on every head of this PR (5 of 5, most recently run 41738536260). It re-runs each of the 6 changed files 10 times. The 5 engine files take about 24 minutes (the differential file alone about 14), so this file never gets its turn, and its multi-worker render is never checked under CPU contention. Any later PR that touches these files will hit the same limit. The gate isn't part of the required Test aggregate, so this doesn't block merge. Either fit the PR inside the gate's budget (fewer or cheaper browser cases) or say in the description that this file isn't flake-gated.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ures have no moving times Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the shutter average inside it Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…est checks the reference frame Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e the trail Renders five blur fixtures through the real producer as a PNG sequence with 4 workers (contiguous blocks, three of them starting mid-film, the #6967 condition): blur off twice, blur on once. Outside the trail of everything that moved, blur on must equal blur off in every frame with two neighbours. The callbacks and scroll-list known differences are pinned by symptom. Import route: a relative import of blurDifferential.ts, fixtures.ts and thresholds.ts from packages/engine/src/services/__fixtures__. The producer tsconfig excludes test files, so I type-checked the file with a temporary tsconfig (rootDir widened); no error. No package subpath was needed. RENDER_OUTSIDE_MAX_MAE: render noise is 0 (blur off, twice, bit-identical), the rewind mutation measured 111.085 when caught, so the 1.0 floor applies. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…calls With 4 workers the blocks started at 0, 15, 30, 45, each on a call, so the rewind mutation showed in a render only 3 times in 5. Seven workers over 60 frames start at 0, 9, 18, 27, 36, 45 and 54, two of them strictly between the calls at frames 16 and 30; the mutation then fails callbacks in 8 of 8 runs (worst MAE 97.02), the clean build in none. A new case checks the partition with distributeFrames. The #list and #label rects now come from the fixture's markup, and MEASURED_RENDER_REWIND_MUTATION is 97.02. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…blindness, fixtures time out Pinned known bugs now require every bad tile inside the pinned region and a worst MAE within half to double of the measured one. tileTouches, Rect and the band check move into blurDifferential.ts with a dpr 2 unit test. The blank-frame census counts canvases it cannot read and asserts none, and names both readings when no blank frame is seen. No console.info on passing runs. Fixtures cases get a 30 s timeout. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he engine helpers The scroll-list loss depends on timing per worker block, so the case now requires frames 2..27 clean and any later bad tile inside #list, without requiring the loss. The callbacks pin keeps requiring its symptom, bounds it to the label and to half to double of its measured size, and its message names the entry to delete. Wording says multi-worker, not 4-worker. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… CI text diff Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… read as blur Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…kew other Chrome tests Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lur render Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nimation frame poll The pin sampled in requestAnimationFrame, so it could miss the gap between the wrap and the first draw. A MutationObserver now reads the output canvas the moment the root is wrapped, which is deterministic, and every fixture must report a wrapped root. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… time Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
9be81b2 to
d50d6c7
Compare
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 0 nits · 0 resolved, 0 open, 0 new · 9be81b2 → d50d6c7
Delta
git range-diff shows all 17 prior commits patch-identical after the rebase. The one new commit raises the Flake gate's timeout-minutes from 30 to 60 (.github/workflows/ci.yml). That matches runsFor in scripts/ci/flake-gate.mjs, which budgets by file count (10 runs each for ≤6 files), with each run capped at 120 s. The gate passed at this head in 34 min. CI is fully green. The Studio timeline viewport gate failed on its first attempt with a thumbnailRenders perf-ratchet miss in code this PR doesn't touch, and passed on rerun.
— Genesis
What
Tests only. Gives us a way to see whether capture motion blur draws the right picture, so we can decide on shipping it default-on. Five bugs reached users that the old tests could not see (worker-start state #6967, nested canvas offset #6975, WebGL freeze #7060, content-box cut and SVG clip path #7080, scroll offset on first wrap). The old tests ask "does blur happen"; these ask "is the picture right".
Two pixel invariants, both against the same composition rendered with blur off:
renderSeek(t, { subFrameDivisions: 24 })). Threshold 36.97 (noise 10.99; smallest catching mutation 124.3).Plus 14 fixtures, a preview liveness test, and a multi-worker render differential through the producer.
Fixtures, the bug each guards, and mutation proof
nested-canvas-offset: fix(core): a canvas in a blurred composition stays where it was laid out #6975. Rule removed: 4376 differing px (max 239) in the canvas region; with the rule, 0. Chrome 153 and 156 identical.callbacks: fix(core): a motion blur worker starting mid-film keeps the right state #6967 rewind.rewindThroughEveryCall()removed: invariant 1 worst tile MAE 106.10 (engine); through the producer with 7 workers 8 of 8 mutated runs fail (97.02), 0 of 6 clean runs.overflow-wordmark,svg-clip-logo: fix(core): keep overflowing content and SVG clip paths in motion-blurred frames #7080 class. Content-box cut mutation 130.13, clip-path mutation 124.32 on invariant 2.scroll-list: scroll offset lost on a fresh first wrap (pinned, see below).webgl-canvas,iframe-in-root,custom-element-in-root,flip-card-3d: cases capture cannot take, which blur through the filter tier.held-card,css-keyframes,stateful-callback,dpr2-mover,portrait-mover: baselines for holds, CSS motion, ripples-style stateful callbacks, device pixel ratio and aspect.Known bugs on main, pinned by symptom
Tests named
KNOWN BUG on main: ...keep the suite green today and flip when the fix lands. A pin requires every bad tile inside the named region and the worst MAE within [0.5x, 2x] of the measured value, so a second defect or a changed symptom fails it.events="on").play()on the capture tier.The producer scroll-list case only bounds the race (zero bad tiles in frames 2-27, any later bad tile inside
#list); the engine pins carry the deterministic "bug fixed" flip.What this does not catch
differscheck and the existing pixels tests cover absence.<video>.Verification
Pins and thresholds were measured on macOS arm64 (Chrome 153 and 156) and then confirmed on Linux CI: three green runs of
Test (runtime)andProducer: integration testswith durations within noise of main (see the comment below). Local: unit 16, differential 73, fixtures + liveness 44, producer differential 18.Review notes: vault page section. Next: adopt Rames Jusso's 60-composition sweep (defects A-O) as fixtures.
🤖 Generated with Claude Code