Skip to content

test(engine): pixel invariants and fixtures that guard default-on capture blur - #7108

Merged
vanceingalls merged 18 commits into
mainfrom
test/motion-blur-default-confidence
Nov 4, 2026
Merged

vanceingalls merged 18 commits into
mainfrom
test/motion-blur-default-confidence

Conversation

@vanceingalls

@vanceingalls vanceingalls commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Tests only. Gives us a way to see whether capture motion blur draws the right picture, so we can decide on shipping it default-on. Five bugs reached users that the old tests could not see (worker-start state #6967, nested canvas offset #6975, WebGL freeze #7060, content-box cut and SVG clip path #7080, scroll offset on first wrap). The old tests ask "does blur happen"; these ask "is the picture right".

Two pixel invariants, both against the same composition rendered with blur off:

  1. Outside the trail, blur-on equals blur-off. Trail mask from blur-off neighbour frames (radius 2 frames), 16 px tile MAE under 1.0. Measured noise 0.0078.
  2. Inside the trail, blur-on is close to the mean of 8 blur-off sub-frame samples across the shutter (renderSeek(t, { subFrameDivisions: 24 })). Threshold 36.97 (noise 10.99; smallest catching mutation 124.3).

Plus 14 fixtures, a preview liveness test, and a multi-worker render differential through the producer.

Fixtures, the bug each guards, and mutation proof

Known bugs on main, pinned by symptom

Tests named KNOWN BUG on main: ... keep the suite green today and flip when the fix lands. A pin requires every bad tile inside the named region and the worst MAE within [0.5x, 2x] of the measured value, so a second defect or a changed symptom fails it.

The producer scroll-list case only bounds the race (zero bad tiles in frames 2-27, any later bad tile inside #list); the engine pins carry the deterministic "bug fixed" flip.

What this does not catch

  • Blur that is missing or too weak: invariant 2's max-tile statistic passes a wrong shutter angle; the fixtures test's differs check and the existing pixels tests cover absence.
  • A DOM change within 2 frames of a checkpoint (masked); checkpoints sit 3 frames from changes.
  • Performance (these check pixels; perf(core): a render frame with no mover to blur skips capture blur #7038's lack of speedup would not show), Windows, Safari, Studio-embedded player, <video>.

Verification

Pins and thresholds were measured on macOS arm64 (Chrome 153 and 156) and then confirmed on Linux CI: three green runs of Test (runtime) and Producer: integration tests with durations within noise of main (see the comment below). Local: unit 16, differential 73, fixtures + liveness 44, producer differential 18.

Review notes: vault page section. Next: adopt Rames Jusso's 60-composition sweep (defects A-O) as fixtures.

🤖 Generated with Claude Code

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 1 important · 1 nit · head ab76d6d

The first Linux run is red in both lanes, and every failure is held DOM text on a capture-tier fixture. Details are inline on thresholds.ts. The other two are timing and dead-field issues.

CI evidence and durations (run 41711675131)
  • Test (runtime): 9 of 73 differential cases fail: callbacks, stateful-callback and overflow-wordmark outside cases at every checkpoint. Worst tiles have MAE 1.8–10 on the text rows. The pixels (18), fixtures (29) and liveness (15) files pass, and supported was true, so the suite really ran. Job time is 8m08s against a 10-minute cap; recent main runs take about 6.5–7 min. The differential file adds 48.6 s, liveness 13.5 s and fixtures 8.4 s.
  • Producer: integration tests: 4m38s against a 20-minute cap. Three blur cases fail with the same signature: overflow-wordmark held frames 13–17 (MAE 2.2–3.2), callbacks frames 18–22 on the label (4.2–6.0) and scroll-list frames 2–6 in the list (2.2–7.2). The blur off twice noise cases pass, so this is not render nondeterminism. The 4th failure (coreRuntimeBrowser.test.ts memory, 4695 vs 800) is in a file this PR doesn't touch; I haven't checked whether it's flaky on main.
  • Shutter centring in shutterTimes matches frames.ts:125 (t - shutter/2).

Audited: all 11 files.

— Genesis

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 109 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

Unstable (2)

  • seqrepeat-none-px-r0-root-z100: tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqplainkeys-none-px-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@vanceingalls
vanceingalls force-pushed the test/motion-blur-default-confidence branch from 6a36ed9 to 1d1ab8b Compare November 3, 2026 21:23
@vanceingalls

Copy link
Copy Markdown
Collaborator Author

Linux CI results for this PR.

Three green runs of both lanes (Chrome 153 pin, ubuntu-latest):

  • Test (runtime): 6m44s, 7m54s, 5m40s (main: 6m24s)
  • Producer: integration tests: 5m28s, 5m51s, 6m19s (main: 5m57s)

What the first Linux run found, all fixed on this branch:

  • The runner draws DOM text with subpixel (LCD) fringes and the capture canvas draws it gray, so every text tile differed by MAE 2–10 on Linux and Windows while macOS measured 0.008. Printed tiles from the runner showed chroma 131–145 on the blur-off arm and 10–26 on the capture arm. That is a host setting, not blur, so both arms now draw grayscale text (--disable-lcd-text in the engine launch; a fontconfig rgba=none override for the producer render). Thresholds are unchanged.
  • The blur test helpers were shipping in the engine tarball and pushed it over its size budget; they are now excluded in files.
  • The 7-Chrome render in the producer lane skewed the Chrome node-count assertions in coreRuntimeBrowser.test.ts (failed 2 of 3 early runs), so that file now runs alone after the rest, and the reachability runner hash is re-pinned.

Not from this PR: Test (support) (and the aggregate Test) fail on current main content too: scripts/check-core-skills.test.mjs expects "ask once for that total" but the Reverse skill text from 341fdd3 says otherwise. Regression shard 10 failed once with an FFmpeg encode error in sub-composition-video on the first run and passed after the rebase.

Not covered yet, as stated in the description: the shutter-angle and weak-blur cases, performance, Windows rendering, and Rames's 60 compositions.

@vanceingalls

Copy link
Copy Markdown
Collaborator Author

All three addressed (head 3438bb1/9be81b2, after the earlier commits):

  • blocker (Linux text): it was host LCD text. A tile dump from the runner showed chroma 131–145 on the blur-off arm and 10–26 on the capture arm, so the DOM draws subpixel fringes and the capture canvas draws gray. Both arms now draw grayscale text: --disable-lcd-text in the engine launch, a fontconfig rgba=none override in the producer render. Thresholds unchanged. Three green runs of Test (runtime) and Producer: integration tests (5m40–7m54 and 5m28–6m19; main 6m24 and 5m57). The coreRuntimeBrowser failure was real interference: it failed 2 of 3 early runs, so the blur render now runs alone after the other files.
  • important (liveness pin): replaced the rAF sampler with a MutationObserver that reads the output canvas the moment the root is wrapped. It detects the blank on all 10 capture fixtures in 8 of 8 local runs. Guards: unreadable == 0, wrapped == 10, blank > 0. It pins the structural state (output empty at the wrap), not a presented frame count, which the trace showed is vsync-dependent.
  • nit (blurs): field and no-op filter removed; the PR description no longer claims a blocked-fixture path.

Also pushed: the blur helpers are out of the engine tarball (size budget), and the reachability runner hash is re-pinned. Test (support) is red on main's own skill text (check-core-skills.test.mjs vs 341fdd3), not this PR.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 1 important · 0 nits · 3 resolved, 0 open, 1 new · ab76d6d → 9be81b2

— Genesis

});
}

describe.skipIf(!HAS_FFMPEG)("page motion blur through a multi-worker render", () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important: Flake gate hits its 30-minute job limit and gets cancelled on every head of this PR (5 of 5, most recently run 41738536260). It re-runs each of the 6 changed files 10 times. The 5 engine files take about 24 minutes (the differential file alone about 14), so this file never gets its turn, and its multi-worker render is never checked under CPU contention. Any later PR that touches these files will hit the same limit. The gate isn't part of the required Test aggregate, so this doesn't block merge. Either fit the PR inside the gate's budget (fewer or cheaper browser cases) or say in the description that this file isn't flake-gated.

vanceingalls and others added 18 commits November 3, 2026 21:27
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ures have no moving times

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the shutter average inside it

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…est checks the reference frame

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e the trail

Renders five blur fixtures through the real producer as a PNG sequence with 4
workers (contiguous blocks, three of them starting mid-film, the #6967
condition): blur off twice, blur on once. Outside the trail of everything that
moved, blur on must equal blur off in every frame with two neighbours. The
callbacks and scroll-list known differences are pinned by symptom.

Import route: a relative import of blurDifferential.ts, fixtures.ts and
thresholds.ts from packages/engine/src/services/__fixtures__. The producer
tsconfig excludes test files, so I type-checked the file with a temporary
tsconfig (rootDir widened); no error. No package subpath was needed.

RENDER_OUTSIDE_MAX_MAE: render noise is 0 (blur off, twice, bit-identical), the
rewind mutation measured 111.085 when caught, so the 1.0 floor applies.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…calls

With 4 workers the blocks started at 0, 15, 30, 45, each on a call, so the
rewind mutation showed in a render only 3 times in 5. Seven workers over 60
frames start at 0, 9, 18, 27, 36, 45 and 54, two of them strictly between the
calls at frames 16 and 30; the mutation then fails callbacks in 8 of 8 runs
(worst MAE 97.02), the clean build in none.

A new case checks the partition with distributeFrames. The #list and #label
rects now come from the fixture's markup, and MEASURED_RENDER_REWIND_MUTATION
is 97.02.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…blindness, fixtures time out

Pinned known bugs now require every bad tile inside the pinned region and a worst
MAE within half to double of the measured one. tileTouches, Rect and the band check
move into blurDifferential.ts with a dpr 2 unit test. The blank-frame census counts
canvases it cannot read and asserts none, and names both readings when no blank frame
is seen. No console.info on passing runs. Fixtures cases get a 30 s timeout.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he engine helpers

The scroll-list loss depends on timing per worker block, so the case now requires
frames 2..27 clean and any later bad tile inside #list, without requiring the loss.
The callbacks pin keeps requiring its symptom, bounds it to the label and to half to
double of its measured size, and its message names the entry to delete. Wording says
multi-worker, not 4-worker.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… CI text diff

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… read as blur

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…kew other Chrome tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lur render

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nimation frame poll

The pin sampled in requestAnimationFrame, so it could miss the gap between the
wrap and the first draw. A MutationObserver now reads the output canvas the
moment the root is wrapped, which is deterministic, and every fixture must
report a wrapped root.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… time

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vanceingalls
vanceingalls force-pushed the test/motion-blur-default-confidence branch from 9be81b2 to d50d6c7 Compare November 4, 2026 05:28

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 0 important · 0 nits · 0 resolved, 0 open, 0 new · 9be81b2 → d50d6c7

Delta

git range-diff shows all 17 prior commits patch-identical after the rebase. The one new commit raises the Flake gate's timeout-minutes from 30 to 60 (.github/workflows/ci.yml). That matches runsFor in scripts/ci/flake-gate.mjs, which budgets by file count (10 runs each for ≤6 files), with each run capped at 120 s. The gate passed at this head in 34 min. CI is fully green. The Studio timeline viewport gate failed on its first attempt with a thumbnailRenders perf-ratchet miss in code this PR doesn't touch, and passed on rerun.

— Genesis

@vanceingalls
vanceingalls added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit d36cad7 Nov 4, 2026
147 of 148 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant