We want to set up visual diff testing as a useful tool to answer 'what if' questions about wikitext changes, i.e. we want to be able to change wikitext parsing and then see how rendering of this changes compared to standard rendering of the same page on the production cluster.
This requires the following to happen:
- Custom mediawiki install that has the desired patches that we want to test.
- Generate an XML dump (wikitext and all associated templates) for a corpus of revisions from the desired wikis.
- Use the XML dump to init the database of the mediawiki install.
- Visual differ configured to fetch the revisions from the custom mediawiki install and the corresponding production wiki (for proper comparison, the screenshots should be taken after running JS code on both fetched pages to expand all auto-collapsed boxes, sections, etc.)
We could potentially use labs instances to run these tests or acquire a dedicated server. In either case, this setup will need to be puppetized.