Skip to content

[flutter_test] Allow maxDifferentPixelsRate tolerance in matchesReferenceImage - #195624

Closed
andywolff wants to merge 1 commit into
flutter:masterfrom
andywolff:matches-reference-image-tolerance
Closed

andywolff wants to merge 1 commit into
flutter:masterfrom
andywolff:matches-reference-image-tolerance

Conversation

@andywolff

@andywolff andywolff commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Part of #193689
Pre-requisite for #192937

This PR adds an optional maxDifferentPixelsRate tolerance parameter to matchesReferenceImage in package:flutter_test and configures a small tolerance for Impeller in transform_test.dart.

We did the same thing for engine/src/flutter/testing/dart/goldens.dart's fuzzyCompareImages in #193937. Can't share because that's in the engine code, which can't depend on framework packages, and matchesReferenceImage is in package:flutter_test.

The test 3D transform renders the same with or without needsCompositing was originally added in #61779 to prevent visual jumping regressions (#41654) where layer integer coordinate snapping caused 3D transforms to wiggle when needsCompositing changed. Across a loop of angles, it compares a raster snapshot of the transform rendered with needsCompositing: true against the same transform rendered with needsCompositing: false using matchesReferenceImage.

Under Skia, both composited and non-composited paths render identically with 0 differing pixels through software rasterization.

Under Impeller, they actually don't render identically because they take different paths:
with compositing: the child is drawn into an offscreen texture and composited as a 3D texture quad sampled via bilinear filtering.
Without composition: the canvas renders vector geometry directly under the 3D perspective matrix using MSAA edge coverage.

Because of this difference, there are antialiasing differences along angled polygon boundaries (consistently ~241 to 247 pixels out of 480,000, or ~0.05% of pixels). It's not a full pixel shift like the issue for which this test was added.

But why do these differences only appear with #192937? Because the host buffer wasn't being reset between snapshots, it actually wasn't re-rendering(?).

I previously attempted to mitigate this in #193936 by reducing loop density (increasing the angle step from 0.01 to 0.05), because I thought it was a problem happening towards the end of the loop after too many iterations. But this didn't help. Now that we have a proper diff tolerance, this PR reverts that angle step back to 0.01, restoring the test to its original angular sampling density.

Pre-launch Checklist

@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label Oct 29, 2026
@github-actions github-actions Bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Oct 29, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a maxDifferentPixelsRate parameter to matchesReferenceImage to allow a configurable tolerance rate for pixel differences during reference image comparisons, and updates relevant tests. Feedback suggests returning null early when countDifferentPixels is 0 to prevent potential division-by-zero or NaN issues when calculating the pixel difference rate.

@andywolff
andywolff marked this pull request as draft October 29, 2026 22:33
@andywolff
andywolff force-pushed the matches-reference-image-tolerance branch from 88abd2d to 97978e1 Compare October 30, 2026 21:47
@andywolff

Copy link
Copy Markdown
Contributor Author

Closing this draft, I found the true root cause of this rendering difference and we can address it without adding a pixel diff tolerance

@andywolff andywolff closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: tests "flutter test", flutter_test, or one of our tests CICD Run CI/CD framework flutter/packages/flutter repository. See also f: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant