The browser tests on beta should be stable, so that they can be used to alert people in case of regressions: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/
This is even more important as it's not possible to subscribe to only failures of the test.wikidata.org browser tests right now (which I would like to have, for post-deploy checking).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • hashar | T144886 Make browsertests pass on beta/ make stable | |||
| Resolved | Tobi_WMDE_SW | T150739 Don't run authority control gadget browsertest with debug=true on beta.wikidata |
Since the same test set runs almost stable on test.wikidata.org I think this is a platform problem - beta being unstable or unavailable during the 2 hours the tests run.
One idea that was just coming to my mind - we could try running a larger test set every time we have a new build for https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/Wikidata directly on integration. Not sure entirely if that is possible.
That sounds ok to me, especially if done post merge.
But still, there should be a way to subscribe people to the results of the tests on test.wikidata.org (and we need to make sure they run after the deploy and early enough so that we can actually fix things before our code hits Wikidata.org).
This will probably be resolved with T167432: Run Wikibase daily browser tests on Jenkins.