The Qunit and Selenium tests take a while, we should extract them to a standalone job and have the others to skip qunit and selenium.
We also run them against all versions of PHP. Probably we should only care of php7.2 at this point? Suggested by Timo iirc.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min | |||
| Resolved | hashar | T232626 CI mediawiki/core run times have increased since July 26th, 2019 | |||
| Resolved | hashar | T232759 Move CI selenium/qunit tests of mediawiki repository to a standalone job | |||
| Resolved | hashar | T249845 CI jobs for WMF deployed extensions should use vendor.git , not composer | |||
| Resolved | • Mholloway | T237596 Add MachineVision dependencies to vendor | |||
| Declined | Reedy | T237588 Security review for MachineVision libraries |
+1 for a separate job that runs qunit+wdio.
I don't think it's worth giving qunit its own job because for MediaWiki core the whole task takes less than 5 seconds. In the shared gate job with all of Wikibase+VE etc installed it takes 25 seconds in total. (Last two random jobs with these numbers: mediawiki-core-quibble#qunit, and wmf-quibble-gate#qunit).
Change 539630 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Dedicated Selenium job for gated repos
Change 539630 merged by jenkins-bot:
[integration/config@master] Dedicated Selenium job for gated repos
For the record, using cold caches:
| wmf-quibble-selenium-hhvm-docker | SUCCESS | 15m 14s |
| wmf-quibble-selenium-php72-docker | SUCCESS | 10m 59s |
@hashar To confirm, we need only one of those, right? Afaik we don't need to cover server variants of DB or PHP for front-end tests.
Nice that php72 is faster. That makes the choice easier :D
Mentioned in SAL (#wikimedia-releng) [2019-10-02T03:07:31Z] <Krinkle> Fix "setup quibble mw-install" Jenkins Console section to not eat all output on new quibble-selenium jobs (T232759)
I have only extracted it for the wmf-quibble* jobs which are for core/extensions/skins gated together and just for master and wmf branches.
The regular quibble jobs would benefit from it as well. Eg for a Wikibase change we have:
| Job | Duration |
|---|---|
| mwgate-node10-docker | 4m 15s |
| quibble-vendor-mysql-php72-docker | 32m 09s (also run selenium tests) |
| quibble-vendor-mysql-php73-docker | 34m 56s (also run selenium tests) |
| wmf-quibble-vendor-mysql-php72-docker | 11m 06s (no more run selenium) |
| wmf-quibble-selenium-php72-docker | 12m 33s (split from above as a standalone job) |
| mwext-php72-phan-docker | 1m 02s |
| mwext-node10-rundoc-docker | 1m 51s |
| mwext-php72-phan-seccheck-docker | 2m 19s |
| mwselenium-quibble-docker | 3m 12s |
| wikibase-client-docker | 2m 23s |
| wikibase-repo-docker | 4m 30s |
The reason I have not split Selenium tests out of quibble-vendor-mysql-php72-docker yet is that it is usually fast. Until it is triggered by a repository having lot of extensions with selenium tests. So probably we just have to split that for the biggest repositories, specially Wikibase which could most probably benefit from having dedicated / finely tuned jobs instead of the generic ones.
Yeah, the separation is most useful for the wmf-quibble job ("extension gate"). For others we would not be saving time as they are not the longest job in the pipeline.
Wikibase indeed could still benefit from it. Although I do wonder if the gate vs non-gate variants are useful for extensions that are in the gate. Especially given the number of dependencies Wikibase has, it might be just as good (or better) to not have the non-gate variant run there at all? Anyway, that might be a separate matter. Splitting seems useful either way. So the task remains open for splitting Wikibase's non-gate job?
Change 543100 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Standalone Selenium jobs for mediawiki/core
Change 543101 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Skip selenium for mediawiki/core (run in standalone job)
Change 543100 merged by jenkins-bot:
[integration/config@master] Standalone Selenium jobs for mediawiki/core
Change 543101 merged by jenkins-bot:
[integration/config@master] Skip selenium for mediawiki/core (run in standalone job)
Change 545580 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Standalone Selenium jobs for extensions/skins
Change 545580 merged by jenkins-bot:
[integration/config@master] Standalone Selenium jobs for extensions/skins
Mentioned in SAL (#wikimedia-releng) [2019-10-23T17:00:53Z] <hashar> Reloading Zuul for https://gerrit.wikimedia.org/r/545580 Standalone Selenium jobs for Wikibase # T232759
For a Wikibase change on master:
| Job | Duration |
|---|---|
| quibble-vendor-mysql-php72-noselenium-docker | 15m 08s |
| quibble-vendor-selenium-docker | 12m 01s |
| wmf-quibble-vendor-mysql-php72-docker | 8m 44s |
| wmf-quibble-selenium-php72-docker | 13m 06s |
That is better than almost half an hour!
Change 558126 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Split Selenium jobs on Wikibase repositories
Change 558126 merged by jenkins-bot:
[integration/config@master] Split Selenium jobs on Wikibase repositories
I have forgot about this task, it is not completed yet. For Wikibase, the Selenium job has been split, but any extension that depends on Wikibase would still run all those tests. So we gotta split it for those repos as well. An example is MachineVision.
Change 587750 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Wikibase Selenium tests to their own job
@Cparle, @matthiasmullie, that follow up the MachineVision deployment you have done today for which I noticed the CI job is super slow. The reason is that we run all extensions tests serially in a single job and that includes the Wikibase selenium tests. This task was to split those tests to a standalone job (and thus run tests in parallel), but it has only be split for Wikibase* repository. Any repository that depends on Wikibase (such as MachineVision), should have the same split.
;)
Some of those repositories are Wikimedia deployed but use a composer based CI job. I have filed T249845 to align them.
Change 587750 merged by jenkins-bot:
[integration/config@master] tests: ensure Wikibase selenium tests are standalone
I think it would be useful to run the phpunit-unit and composer-test checks as part of the "Selenium only" jobs to get faster feedback on a failing patch. For example in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/673218/2#message-95b611105aca937faeb551300a4d5a2f68cd6df4 I could have known after ~2 minutes that the patch needed adjustment, instead I found out 13 minutes later when the Selenium job finished running.
I strongly disagree. Selenium tests are already our slowest test job. Adding duplicate tests in front of them will slow down every merge and test for everyone. I'd rather we worked on async result responses from zuul, but I believe that's not possible sadly, and the whole mess is going to be replaced when we move to GitLab anyway. :-(
True, but by the time we're ready to run the Selenium tests, we have everything we need to run the unit tests for core/extensions/skins and for core that takes 10 seconds. Likewise, running composer test takes a couple of seconds (https://integration.wikimedia.org/ci/job/mediawiki-quibble-composertest-php72-docker/42079/console#console-section-6). From a theoretical standpoint, yes I'd rather have async feedback and not mix/match the tests but from a practical point of view I'd rather know within 2 minutes that my patch is never going to be mergeable due to failing unit tests or php lint / codestyle issues rather than wait 15 minutes for Selenium to do its thing.