This'll require the following infrastructure:
- A bot that listens whatever Gerrit API there is for new patches, runs a suite of browser tests, and reports the results
- A server that we run the browser tests on, which hosts an instance of MediaWiki with all the appropriate extensions installed
We've talked with releng and we know that they are working on a generic solution, but it will take a lot of time. We need a temporary solution so we knowingly accept the cost of maintaining our own browser testing infrastructure but we feel that it's far outweighed by the increased confidence in our code. (Our products are very user exposed and we need a good integration tests story).
- Browser tests are run on MobileFrontend and Gather on each patchset and get a report of the failing browser tests.
- A testing environment is set up with mediawiki and our extensions, and the appropriate scripts for triggering the tests.
- What happens in the testing environment is easily accessible by engineers (transparency about what is going on) (log everything and expose logs).