Hmm this was really annoying. I've been trying to backtrack/understand how this works. I've been trying with two images that is generated with a diff with VisualMetrics:

If I test them in a tool without any fuzz I can see small diff (the red):

However when I run with fuzz factor 10% (as Visual Metrics) there's no diff.

I've tried it manually:

docker run -v “$(pwd)“:/video sitespeedio/visualmetrics-deps convert “/video/ms_006367.png” “/video/ms_006450.png” -crop 241x372+0+8 miff:- | compare -metric AE - -fuzz 10% null:

And I get a 0 back (0 pixels diff). So Visual Metrics shouldn't say it diff ... let me see if I can recreate the full video locally.

Adding an example file. Running default visual change gives last visual change something like 11 seconds but the "real" one should be more like 6 seconds.

There's two ways to fix this: when visual metrics removes frames from the end, it uses a fuzz factor of 10% (that's default that is used all over) and then compares pixel by pixel and the there must be zero pixels diff, for making sure that the screenshots are the same. However when looking for Visual Elements, the element also uses a 10 % fuzz but a 2% of pixels that can differs.

I've tried changing both and with that test file a 13% fuzz fixes the problem or a pixels diff of 0.01%. I think adding that pixel diff is better so we keep the same fuzz factor on all places. I'll make the change later today and push it on Monday.

I've pushed the fix now (with an annotation), I'll check that it really works before I close it.

No that didn't help, there is still something like 13 s last visual change happening for Obama on mobile. I've changed so that we look a 0.01% of the pixels can differ (that worked locally) but it seems that the diff is too small. Let me increase it a little and try again tomorrow.

I did some more changes and now uses 0.05% diff that's ok, that made it work so it runs in the servers now. Let me have a look tomorrow that everything is ok and then close.

Looks much better!