The easiest way to use visual regression tests is to have a computer run them for you automatically. This task proposes adding a non-blocking CI job to the Vector repo that runs on each patch set made to Vector. If that goes well, other repos could be added.
It would run something like the following which would compare master against the change on top of master:
./pixel.js reference && ./pixel.js -c <Change-Id of Gerrit Patch>
The preceding command launches several Docker containers (e.g. visual regression test container, a container with MediaWiki, a container with the database) and runs the visual regression tests. If that command exits with 0, the job passes. If it exits with something else, it fails. However, failures shouldn't block merging and should only be seen as flags that need discretionary review.
- For each Gerrit patch set made to Vector, CI runs the command above which takes a visual diff of master against the change/changes on top of master.
- The job should provide a publicly accessible link to the report generated (e.g. see https://pixel.wmcloud.org/desktop/ for example) where one can review the screenshots taken.
- If the change includes dependencies (Depends-On) that Pixel doesn't support, the job can be skipped.