This is something I've been experimenting with in spare time for a while.

The idea is: Render a visual diff for one or more pages and states thereof (comparing the result of the current change to the result of the latest master, or whatever target branch the commit has).

The screenshots would be created using PhantomJS' render() API. Or, while we don't need cross-browser per se, having a browser more representative than PhantomJS would be nice. Perhaps using Chromium under Xvfb with a large enough window (we don't need it to be very tall). And capture output from Xvfb using ImageMagic display import.

Rough idea for the Jenkins job:

• Run project setup (e.g. build script for projects like OOjs UI and VisualEditor; installing MediaWiki for core/extensions). Then expose workspace to the local web server (We've got re-usable macros for this already).

• Run the scenarios or urls for the current project and capture the screen after each scenario.

• Compare them against the ones from the last run (e.g. for a commit to master, compare them to the latest master build). TODO: Will need to be stored somewhere. Shared NFS maybe? store/{project}/{branch}.

• In test pipeline:
  • If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts in Jenkins. Otherwise delete the image.
• In the post-merge pipeline:
  • Replace the images in the store with those of this build.

Scenarios:

I imagine we'll need to support two kinds of scenarios:

• Plain url.
• Web driver steps (for large interfaces not accessible by url). This should *not* be used to trigger every possible dialog and component, that slows the test matrix and only tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page. Use these two assert the composition rather.

A few urls we might want:

• mediawiki-core:
  • /index.php?title=Main_Page
  • /index.php?title=Main_Page&useskin=monobook
  • /index.php?title=Main_Page&action=edit
  • /index.php?title=Main_Page&action=history
  • /index.php?title=Special:UserLogin
  • /index.php?title=Special:UserLogin/signup
  • /index.php?title=Special:Search&search=wiki
• VisualEditor:
  • /demos/ve/#!/src/pages/empty.html
  • /demos/ve/#!/src/pages/simple.html
  • /demos/ve/#!/src/pages/complex.html
• oojs-ui:
  • /demos/icons.html
  • /demos/widgets.html

A few implementations that exist:

• https://github.com/uber/image-diff/
• https://github.com/bslatkin/dpxdt/
• Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc
• Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc

Behind these is basically just a ImageMagick compare command between two PNGs.

• https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127
• https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65
• http://www.imagemagick.org/Usage/compare/

`
compare
      -verbose
      -metric RMSE
      -highlight-color RED
      -compose Src
      mytest-latest.png
      mytest-build.png
      mytest-diff.png

See also:

• T64511: Mass rendering tests / visual diff