This is something I've been experimenting with in spare time for a while.
The idea is: Render a visual diff for one or more pages and states thereof (comparing the result of the current change to the result of the latest master, or whatever target branch the commit has).
The PNGs would be created using PhantomJS' render() API.
Jenkins job macro for this:
* For core and extensions, this would be installing MediaWiki and exposing it in the local Apache, for client-side libraries like VisualEditor and OOjs UI, this is just running a build script and exposing their demo/ directory in the local Apache
* Create PNGs for all the configures urls (relative paths to their webroot, we'd prefix our apache server + unique subdirectory to this).
* Compare them against the ones from the previous post-merge job (stores somewhere persistently by repository/branch/url).
* If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts (we should clear them after some time as they'll grow quickly, as optimisation we can delete them within the job if there is no diff).
From the post-merge job:
* Run the main macro.
* Copy the created screenshots to the persistent "latest" directory for comparison by the next unmerged patch set test.
There are many different tools, but from a result perspective I imagine it needs to be support the following axis:
* Input:
- By url (Just a plain url, e.g. `{localwiki}/wiki/Main_Page`, `{localwiki}/wiki/Special:UserLogin`, `action=edit` etc.)
- By webdriver steps (for large interfaces not accessible by url. This should *not* be used to trigger every possible dialog and component, that explodes the test matrix and only repeats tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page.)
* Output:
- Capture browser screenshot for a set of browsers.
--------
A few example urls for different repositories that we would want visual regression testing.
mediawiki-core:
/index.php?title=Main_Page
/index.php?title=Main_Page&useskin=monobook
/index.php?title=Main_Page&action=edit
/index.php?title=Main_Page&action=history
/index.php?title=Special:UserLogin
/index.php?title=Special:UserLogin/signup
/index.php?title=Special:Search&search=wiki
VisualEditor:
/demos/ve/#!/src/pages/empty.html
/demos/ve/#!/src/pages/simple.html
/demos/ve/#!/src/pages/complex.html
oojs-ui:
/demos/icons.html
/demos/widgets.html
A few implementations that exist:
* https://github.com/uber/image-diff/
* https://github.com/bslatkin/dpxdt/
* Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc
* Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc
Behind these is basically just a ImageMagick compare command between two PNGs.
* https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127
* https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65
* http://www.imagemagick.org/Usage/compare/
````
compare
-verbose
-metric RMSE
-highlight-color RED
-compose Src
mytest-latest.png
mytest-build.png
mytest-diff.png
```
See also:
* {T64511}