Allow color delta tolerance in in-memory fuzzy image comparator by andywolff · Pull Request #193937 · flutter/flutter

andywolff · 2026-10-06T22:37:07Z

Part of #193029
Pre-requisite for #192937

This PR adds color delta and pixel rate tolerance parameters to the engine's in-memory ImageComparer.fuzzyCompareImages helper, and updates Canvas.drawParagraph renders tab as space instead of tofu to tolerate minor subpixel anti-aliasing variations across consecutive passes.

While verifying #192937, Linux linux_host_engine_test failed repeatedly in canvas_test.dart under Vulkan with SwiftShader.

The test Canvas.drawParagraph renders tab as space instead of tofu renders >\t< and > < into separate in-memory images via toImage and compares them using comparer.fuzzyCompareImages(tabImage, spaceImage). Despite the name fuzzyCompareImages, the previous implementation was pixel exact with no fuzziness at all. How un-comfy!

Under CPU-based software Vulkan drivers (such as SwiftShader on Linux CI runners), anti-aliasing edge rounding differences between separate snapshot passes produce 1–2 channel delta variations on glyph boundaries, causing the comparison to fail.

Previously, consecutive snapshots left un-cleared host buffer data behind that masked subpixel variations. With my leak fix PR, resetting the buffer on each snapshot forces every text pass to allocate from scratch, exposing small driver-level raster differences on software Vulkan.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

gemini-code-assist

Code Review

This pull request introduces tolerance thresholds (maxColorDelta and maxDifferentPixelsRate) to the fuzzyCompareImages method in goldens.dart to allow for minor color and pixel variations in image comparisons, and applies a small color delta in canvas_test.dart. The review feedback suggests improving safety by handling potential null values from toByteData() and optimizing performance in the comparison loops by using Uint32List and Uint8List instead of ByteData getters.

gaaclarke · 2026-10-09T21:59:05Z

-      for (var x = 0; x < golden.width; x++) {
-        if (getPixel(goldenData, x, y) != getPixel(testImageData, x, y)) {
+    final ByteData? goldenData = await golden.toByteData();
+    final ByteData? testImageData = await testImage.toByteData();


Do we have to care about wide gamut images? If so we'll need to grab the byte data as doubles.

If I'm reading the code correctly, all existing callers of fuzzyCompareImages only deal with 8-bit sRGB images. It looks like skia goldctl only supports 8-bit images. We have some Impeller C++ wide gamut tests with 10-bit images, but it looks like MetalGoldenScreenshotter converts them down to 8-bit PNGs before sending them to Skia Gold. Any actual 10-bit pixel verification we do skips skia gold and checks rawExtendedRgba128 directly without image comparators (example: high_bitrate_texture_test.dart). If we want wide gamut in skia gold, I suspect we'll need to add support to the goldctl upload flow first.

Considering that, I added an assertion and a doc comment requiring ColorSpace.sRGB for now just to make this super clear.

gaaclarke · 2026-10-09T22:00:42Z

    return true;
  }
+
+  static int _maxChannelDelta(Uint8List a, Uint8List b, int offset) {


Would it make more sense to compare distances between vec3 values?

Done, and changed the threshold param and callsites to use double

gaaclarke

lgtm! thanks

auto-submit · 2026-10-14T01:05:04Z

autosubmit label was removed for flutter/flutter/193937, because - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.

Roll Flutter from 4249ccbc18a3 to 426be968b5eb (44 revisions) flutter/flutter@4249ccb...426be96 2026-10-14 awolff@google.com Allow color delta tolerance in in-memory fuzzy image comparator (flutter/flutter#193937) 2026-10-14 zhongliu88889@gmail.com [web] Drop stale composing range after text is replaced mid-composition (flutter/flutter#194096) 2026-10-14 engine-flutter-autoroll@skia.org Roll Skia from 2a2bbea66844 to acde3755d2e2 (1 revision) (flutter/flutter#194376) 2026-10-14 engine-flutter-autoroll@skia.org Roll Skia from 128c983b12de to 2a2bbea66844 (1 revision) (flutter/flutter#194371) 2026-10-14 engine-flutter-autoroll@skia.org Roll Skia from 48a294ed4955 to 128c983b12de (3 revisions) (flutter/flutter#194353) 2026-10-14 kevmoo@users.noreply.github.com feat(flutter_tools): extract WebDebugConfig from DebuggingOptions (flutter/flutter#194312) 2026-10-14 engine-flutter-autoroll@skia.org Roll Skia from ea6781c5b6d1 to 48a294ed4955 (2 revisions) (flutter/flutter#194349) 2026-10-14 chingjun@google.com Include <vector> in Android proc_table.h. (flutter/flutter#194325) 2026-10-14 engine-flutter-autoroll@skia.org Roll Dart SDK from 5933cb486b4a to 6b50116630d0 (28 revisions) (flutter/flutter#194341) 2026-10-14 97480502+b-luk@users.noreply.github.com Move GLES3 instancing API entry points to FOR_EACH_IMPELLER_GLES3_PROC (flutter/flutter#194319) 2026-10-14 kevmoo@users.noreply.github.com feat(flutter_tools): extract AndroidGradleConfig and move toGradleConfig to AndroidBuildInfo (flutter/flutter#194016) 2026-10-14 engine-flutter-autoroll@skia.org Roll Skia from ae297bec6012 to ea6781c5b6d1 (8 revisions) (flutter/flutter#194331) 2026-10-14 bkonyi@google.com [flutter_tools] Migrate gradle_errors and deferred_components_prebuild_validator away from globals (flutter/flutter#193649) 2026-10-14 kevmoo@users.noreply.github.com [Impeller] Cache GLES sampler configuration in RenderPassStateCache (flutter/flutter#193922) 2026-10-14 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#194332) 2026-10-13 bkonyi@google.com [tool] Migrate RunCommand to modular dependency injection (flutter/flutter#190791) 2026-10-13 mdebbar@google.com Include `engine/src/flutter/.ci.yaml` in quarterly `goldctl` checklist (flutter/flutter#194316) 2026-10-13 nshahan@google.com [flutter_tools] Remove the last AMD module system support (flutter/flutter#193607) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from 8b3394a60654 to ae297bec6012 (5 revisions) (flutter/flutter#194310) 2026-10-13 zhongliu88889@gmail.com [web] Remove the a11y placeholder only after the gesture ends (flutter/flutter#194109) 2026-10-13 mdebbar@google.com Upgrade goldctl to the latest version (flutter/flutter#193932) 2026-10-13 bkonyi@google.com [flutter_tools] Lazily initialize AndroidStudio validation and optimize candidate selection (flutter/flutter#191968) 2026-10-13 68429735+Vonarian@users.noreply.github.com [flutter_tools] Clean up outdated widget preview TODOs and refactor static state (flutter/flutter#189437) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from e5b5db6d0d6f to 8b3394a60654 (2 revisions) (flutter/flutter#194306) 2026-10-13 matt.boetger@gmail.com Redistribute TESTOWNERS for Android team (flutter/flutter#193861) 2026-10-13 kevmoo@users.noreply.github.com feat(flutter_tools): introduce Test and Drive specification objects (flutter/flutter#194017) 2026-10-13 santoshakil@users.noreply.github.com [Impeller] Keep the aspect ratio when shrinking images to the max texture size (flutter/flutter#194010) 2026-10-13 pq@users.noreply.github.com [flutter tool] preserve inherited `DASH__SUPPRESS_ANALYTICS=true` for sub-tools (flutter/flutter#194261) 2026-10-13 engine-flutter-autoroll@skia.org Roll Packages from d8d9e97 to a1ed23c (9 revisions) (flutter/flutter#194302) 2026-10-13 manjian2006@gmail.com Use PE/COFF AOT snapshots for Windows x64 (flutter/flutter#187594) 2026-10-13 omar.alshyokh@gmail.com Fix Expansible crash when PageStorage holds a non-bool value (flutter/flutter#192195) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from ffeccb3ca58a to e5b5db6d0d6f (1 revision) (flutter/flutter#194297) 2026-10-13 49402500+fahaddoc@users.noreply.github.com Explain why initState() is too early to depend on an inherited widget (flutter/flutter#192727) 2026-10-13 victorsanniay@gmail.com Refactor TestPlatformDispatcher (flutter/flutter#192747) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from 3de0984208ec to ffeccb3ca58a (2 revisions) (flutter/flutter#194291) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from 0580c2b780f6 to 3de0984208ec (2 revisions) (flutter/flutter#194281) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from f641d55e07a8 to 0580c2b780f6 (3 revisions) (flutter/flutter#194275) 2026-10-13 engine-flutter-autoroll@skia.org Roll Skia from 0e5d8d1be81b to f641d55e07a8 (1 revision) (flutter/flutter#194266) 2026-10-13 victorsanniay@gmail.com Scaffold view metrics overrides (flutter/flutter#192764) 2026-10-13 kevmoo@users.noreply.github.com [web] Add SemanticsRole.listBox and SemanticsRole.option (flutter/flutter#192971) 2026-10-13 15619084+vashworth@users.noreply.github.com Pin mac_unopt to macOS 15 (flutter/flutter#194254) 2026-10-13 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#194257) 2026-10-12 engine-flutter-autoroll@skia.org Roll Skia from 17f6986c226c to 0e5d8d1be81b (4 revisions) (flutter/flutter#194249) 2026-10-12 97480502+b-luk@users.noreply.github.com Make DisplayLists' use of DlPaint, SkPaint, and impeller::Paint agree on the default isAntiAlias (flutter/flutter#193939) If this roll has caused a breakage, revert this CL and set the roller ...

…ter#193937) Part of flutter#193029 Pre-requisite for flutter#192937 This PR adds color delta and pixel rate tolerance parameters to the engine's in-memory `ImageComparer.fuzzyCompareImages` helper, and updates `Canvas.drawParagraph renders tab as space instead of tofu` to tolerate minor subpixel anti-aliasing variations across consecutive passes. While verifying flutter#192937, `Linux linux_host_engine_test` failed repeatedly in `canvas_test.dart` under Vulkan with SwiftShader. The test `Canvas.drawParagraph renders tab as space instead of tofu` renders `>\t<` and `> <` into separate in-memory images via `toImage` and compares them using `comparer.fuzzyCompareImages(tabImage, spaceImage)`. Despite the name `fuzzyCompareImages`, the previous implementation was pixel exact with no fuzziness at all. How un-comfy! Under CPU-based software Vulkan drivers (such as SwiftShader on Linux CI runners), anti-aliasing edge rounding differences between separate snapshot passes produce 1–2 channel delta variations on glyph boundaries, causing the comparison to fail. Previously, consecutive snapshots left un-cleared host buffer data behind that masked subpixel variations. With my leak fix PR, resetting the buffer on each snapshot forces every text pass to allocate from scratch, exposing small driver-level raster differences on software Vulkan. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

andywolff · 2026-11-05T21:42:38Z

Reason for revert: not needed because #196063 is the true root cause of the flakiness

flutteractionsbot · 2026-11-05T21:43:30Z

Successfully created revert PR: #196067

@andywolff

…or (flutter#196067) Reverts: [Allow color delta tolerance in in-memory fuzzy image comparator](flutter#193937) Initiated by: @andywolff Reason for reverting: not needed because flutter#196063 is the true root cause of the flakiness Original PR Author: @andywolff Reviewed By: @gaaclarke The original PR description is provided below: Part of flutter#193029 Pre-requisite for flutter#192937 This PR adds color delta and pixel rate tolerance parameters to the engine's in-memory `ImageComparer.fuzzyCompareImages` helper, and updates `Canvas.drawParagraph renders tab as space instead of tofu` to tolerate minor subpixel anti-aliasing variations across consecutive passes. While verifying flutter#192937, `Linux linux_host_engine_test` failed repeatedly in `canvas_test.dart` under Vulkan with SwiftShader. The test `Canvas.drawParagraph renders tab as space instead of tofu` renders `>\t<` and `> <` into separate in-memory images via `toImage` and compares them using `comparer.fuzzyCompareImages(tabImage, spaceImage)`. Despite the name `fuzzyCompareImages`, the previous implementation was pixel exact with no fuzziness at all. How un-comfy! Under CPU-based software Vulkan drivers (such as SwiftShader on Linux CI runners), anti-aliasing edge rounding differences between separate snapshot passes produce 1–2 channel delta variations on glyph boundaries, causing the comparison to fail. Previously, consecutive snapshots left un-cleared host buffer data behind that masked subpixel variations. With my leak fix PR, resetting the buffer on each snapshot forces every text pass to allocate from scratch, exposing small driver-level raster differences on software Vulkan. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

andywolff added the CICD Run CI/CD label Oct 6, 2026

github-actions Bot added the engine flutter/engine related. See also e: labels. label Oct 6, 2026

Allow color delta tolerance in in-memory fuzzy image comparator

fede63b

andywolff force-pushed the fix-vulkan-fuzzy-compare branch 2 times, most recently from 1925cbe to fede63b Compare October 7, 2026 15:54

andywolff requested a review from gaaclarke October 7, 2026 16:09

andywolff marked this pull request as ready for review October 7, 2026 16:09

gemini-code-assist Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread engine/src/flutter/testing/dart/goldens.dart Resolved

andywolff removed the request for review from gaaclarke October 7, 2026 16:19

andywolff added 2 commits October 7, 2026 09:24

Handle null toByteData and use Uint32List and Uint8List array indexing

32a6be3

Specify type annotation for maxDelta in _maxChannelDelta

0c88492

andywolff requested a review from gaaclarke October 7, 2026 20:30

gaaclarke reviewed Oct 9, 2026

View reviewed changes

Use vec3 Euclidean color distance and assert sRGB in fuzzyCompareImages

f03eaae

andywolff requested a review from gaaclarke October 12, 2026 19:14

gaaclarke approved these changes Oct 12, 2026

View reviewed changes

Merge branch 'master' into fix-vulkan-fuzzy-compare

ecb17e9

andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 13, 2026

auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 14, 2026

andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 14, 2026

auto-submit Bot added this pull request to the merge queue Oct 14, 2026

Merged via the queue into flutter:master with commit 426be96 Oct 14, 2026
30 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 14, 2026

engine-flutter-autoroll mentioned this pull request Oct 14, 2026

Roll Flutter from 4249ccbc18a3 to 426be968b5eb (44 revisions) flutter/packages#13256

Merged

andywolff mentioned this pull request Oct 29, 2026

[flutter_test] Allow maxDifferentPixelsRate tolerance in matchesReferenceImage #195624

Closed

11 tasks done

andywolff mentioned this pull request Nov 5, 2026

[Impeller] Missing SurfaceContextVK submission tracker and double HostBuffer reset #196063

Open

andywolff added the revert Autorevert PR (with "Reason for revert:" comment) label Nov 5, 2026

flutteractionsbot mentioned this pull request Nov 5, 2026

Revert: Allow color delta tolerance in in-memory fuzzy image comparator #196067

Merged

11 tasks done

flutteractionsbot removed the revert Autorevert PR (with "Reason for revert:" comment) label Nov 5, 2026

Navigation Menu

Repository navigation

Allow color delta tolerance in in-memory fuzzy image comparator - #193937

Allow color delta tolerance in in-memory fuzzy image comparator#193937
auto-submit[bot] merged 5 commits into
flutter:masterfrom
andywolff:fix-vulkan-fuzzy-compare

andywolff commented Oct 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gaaclarke Oct 9, 2026

Uh oh!

andywolff Oct 12, 2026

Uh oh!

gaaclarke Oct 9, 2026

Uh oh!

andywolff Oct 12, 2026

Uh oh!

gaaclarke left a comment

Uh oh!

auto-submit Bot commented Oct 14, 2026

Uh oh!

Uh oh!

andywolff commented Nov 5, 2026

Uh oh!

flutteractionsbot commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andywolff commented Oct 6, 2026

Pre-launch Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gaaclarke Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

andywolff Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

gaaclarke Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

andywolff Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

auto-submit Bot commented Oct 14, 2026

Uh oh!

Uh oh!

andywolff commented Nov 5, 2026

Uh oh!

flutteractionsbot commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants