Page MenuHomePhabricator

Allow for expected regressions
Closed, DuplicatePublic

Description

Captured from @Jdlrobson in https://github.com/wikimedia/pixel/issues/107

Quite often we make changes that cause intentional visual regressions. One example is changing the margin of the language button. This often leads to lots of failures, which then hide real visual regressions that occur later on.

I experimented with a fast forward concept in Pixel where you could override latest-branch to mean a certain commit.
See https://github.com/wikimedia/pixel/commit/ab10804bb8cf1f20e157cd7f7a201d69eed49c65 and related patches.

However, something wasn’t quite working (at one point I was testing that commit against itself and another time it was trying to checkout the commit on unrelated repos. I reverted that experiment.

I've come to the conclusion that perhaps the best thing to do in these situations would be to store reference screenshots locally inside Pixel itself for a given release branch.