Change Details

This is something I've been experimenting with in spare time for a while. The idea is: Render a visual diff for one or more pages and states thereof (comparing the result of the current change to the result of the latest master, or whatever target branch the commit has). The PNGs would be created using PhantomJS' render() API. Jenkins job macro for this: * For core and extensions, this would be installing MediaWiki and exposing it in the local Apache, for client-side libraries like VisualEditor and OOjs UI, this is just running a build script and exposing their demo/ directory in the local Apache * Create PNGs for all the configures urls (relative paths to their webroot, we'd prefix our apache server + unique subdirectory to this). * Compare them against the ones from the previous post-merge job (stores somewhere persistently by repository/branch/url). * If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts (we should clear them after some time as they'll grow quickly, as optimisation we can delete them within the job if there is no diff). From the post-merge job: * Run the main macro. * Copy the created screenshots to the persistent "latest" directory for comparison by the next unmerged patch set test. There are many different tools, but from a result perspective I imagine it needs to be support the following axis: * Input: - By url (Just a plain url, e.g. `{localwiki}/wiki/Main_Page`, `{localwiki}/wiki/Special:UserLogin`, `action=edit` etc.) - By webdriver steps (for large interfaces not accessible by url. This should *not* be used to trigger every possible dialog and component, that explodes the test matrix and only repeats tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page.) * Output: - Capture browser screenshot for a set of browsers. -------- A few example urls for different repositories that we would want visual regression testing. mediawiki-core: /index.php?title=Main_Page /index.php?title=Main_Page&useskin=monobook /index.php?title=Main_Page&action=edit /index.php?title=Main_Page&action=history /index.php?title=Special:UserLogin /index.php?title=Special:UserLogin/signup /index.php?title=Special:Search&search=wiki VisualEditor: /demos/ve/#!/src/pages/empty.html /demos/ve/#!/src/pages/simple.html /demos/ve/#!/src/pages/complex.html oojs-ui: /demos/icons.html /demos/widgets.html A few implementations that exist: * https://github.com/uber/image-diff/ * https://github.com/bslatkin/dpxdt/ * Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc * Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc Behind these is basically just a ImageMagick compare command between two PNGs. * https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127 * https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65 * http://www.imagemagick.org/Usage/compare/ ```` compare -verbose -metric RMSE -highlight-color RED -compose Src mytest-latest.png mytest-build.png mytest-diff.png ``` See also: * {T64511}

This is something I've been experimenting with in spare time for a while. The idea is: Render a visual diff for one or more pages and states thereof (comparing the result of the current change to the result of the latest master, or whatever target branch the commit has). The screenshots would be created using PhantomJS' render() API. Or, while we don't need cross-browser per se, having a browser more representative than PhantomJS would be nice. Perhaps using Chromium under Xvfb with a large enough window (we don't need it to be very tall). And capture output from Xvfb using ImageMagic display `import`. Rough idea for the Jenkins job: * Run project setup (e.g. build script for projects like OOjs UI and VisualEditor; installing MediaWiki for core/extensions). Then expose workspace to the local web server (We've got re-usable macros for this already). * Run the scenarios or urls for the current project and capture the screen after each scenario. * Compare them against the ones from the last run (e.g. for a commit to master, compare them to the latest master build). TODO: Will need to be stored somewhere. Shared NFS maybe? `store/{project}/{branch}`. * In test pipeline: * If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts in Jenkins. Otherwise delete the image. * In the post-merge pipeline: * Replace the images in the store with those of this build. Scenarios: I imagine we'll need to support two kinds of scenarios: * Plain url. * Web driver steps (for large interfaces not accessible by url). This should *not* be used to trigger every possible dialog and component, that slows the test matrix and only tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page. Use these two assert the composition rather. A few urls we might want: * mediawiki-core: - /index.php?title=Main_Page - /index.php?title=Main_Page&useskin=monobook - /index.php?title=Main_Page&action=edit - /index.php?title=Main_Page&action=history - /index.php?title=Special:UserLogin - /index.php?title=Special:UserLogin/signup - /index.php?title=Special:Search&search=wiki * VisualEditor: - /demos/ve/#!/src/pages/empty.html - /demos/ve/#!/src/pages/simple.html - /demos/ve/#!/src/pages/complex.html * oojs-ui: - /demos/icons.html - /demos/widgets.html A few implementations that exist: * https://github.com/uber/image-diff/ * https://github.com/bslatkin/dpxdt/ * Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc * Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc Behind these is basically just a ImageMagick compare command between two PNGs. * https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127 * https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65 * http://www.imagemagick.org/Usage/compare/ ```` compare -verbose -metric RMSE -highlight-color RED -compose Src mytest-latest.png mytest-build.png mytest-diff.png ``` See also: * {T64511}

This is something I've been experimenting with in spare time for a while. The idea is: Render a visual diff for one or more pages and states thereof (comparing the result of the current change to the result of the latest master, or whatever target branch the commit has). The PNGsscreenshots would be created using PhantomJS' render() API. Jenkins job macro for this:Or, while we don't need cross-browser per se, having a browser more representative than PhantomJS would be nice. Perhaps using Chromium under Xvfb with a large enough window (we don't need it to be very tall). And capture output from Xvfb using ImageMagic display `import`. * For core and extensions, this would be installing MediaWiki and exposing it in the local Apache, for client-side libraries like VisualEditor and OOjs UI, this is just running a build script and exposing their demo/ directory in the local ApacheRough idea for the Jenkins job: * Create PNGs* Run project setup (e.g. build script for all the configures urls (relative paths to their webroot,projects like OOjs UI and VisualEditor; installing MediaWiki for core/extensions). we'd prefix our apacheThen expose workspace to the local web server + unique subdirectory to(We've got re-usable macros for this already). * Compare them against the ones from* Run the scenarios or urls for the previous post-merge job (stores somewhere persistently by repository/branch/url)current project and capture the screen after each scenario. * If different* Compare them against the ones from the last run (e.g. for a commit to master, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts (we should clear them after some time as they'll grow quickly,compare them to the latest master build). TODO: Will need to be stored somewhere. Shared NFS maybe? as optimisation we can delete them within the job if there is no diff)`store/{project}/{branch}`. From the post-merge job* In test pipeline: * If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts in Jenkins. Otherwise delete the image. * In the post-merge pipeline: * Replace the images in the store with those of this build. Scenarios: * Run the main macroI imagine we'll need to support two kinds of scenarios: * Plain url. * Copy the created screenshots to the persistent "latest" directory for comparison by the next unmerged patch set test. There are many different tools,Web driver steps (for large interfaces not accessible by url). This should *not* be used to trigger every possible dialog and component, that slows the test matrix and only tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page. but from a result perspective I imagine it needs to be support the following axis:Use these two assert the composition rather. * Input:A few urls we might want: - By url (Just a plain url, e.g. `{localwiki}/wiki/Main_Page`, `{localwiki}/wiki/Special:UserLogin`, `action=edit` etc.)* mediawiki-core: - By webdriver steps (for large interfaces not accessible by url. This should *not* be used to trigger every possible dialog and component, that explodes the test matrix and only repeats tests for no reason. More useful would be to capture individual components via e.g. the OOjs UI demo page.) * Output:/index.php?title=Main_Page - Capture browser screenshot for a set of browsers. -------- A few example urls for different repositories that we would want visual regression testing. mediawiki-core:/index.php?title=Main_Page&useskin=monobook - /index.php?title=Main_Page&action=edit - /index.php?title=Main_Page&useskin=monobookaction=history - /index.php?title=Main_Page&action=editSpecial:UserLogin - /index.php?title=Main_Page&action=historySpecial:UserLogin/signup - /index.php?title=Special:UserLoginSearch&search=wiki /index.php?title=Special:UserLogin/signup* VisualEditor: /index.php?title=Special:Search&search=wiki VisualEditor:- /demos/ve/#!/src/pages/empty.html - /demos/ve/#!/src/pages/emptysimple.html - /demos/ve/#!/src/pages/sicomplex.html /demos/ve/#!/src/pages/complex.html * oojs-ui: - /demos/icons.html - /demos/widgets.html A few implementations that exist: * https://github.com/uber/image-diff/ * https://github.com/bslatkin/dpxdt/ * Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc * Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc Behind these is basically just a ImageMagick compare command between two PNGs. * https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127 * https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65 * http://www.imagemagick.org/Usage/compare/ ```` compare -verbose -metric RMSE -highlight-color RED -compose Src mytest-latest.png mytest-build.png mytest-diff.png ``` See also: * {T64511}