Page MenuHomePhabricator

Move CI selenium/qunit tests of mediawiki repository to a standalone job
Open, HighPublic

Description

The Qunit and Selenium tests take a while, we should extract them to a standalone job and have the others to skip qunit and selenium.

We also run them against all versions of PHP. Probably we should only care of php7.2 at this point? Suggested by Timo iirc.

Details

Related Gerrit Patches:

Event Timeline

hashar created this task.Sep 12 2019, 4:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 12 2019, 4:33 PM
Krinkle added a comment.EditedSep 27 2019, 3:35 AM

+1 for a separate job that runs qunit+wdio.

I don't think it's worth giving qunit its own job because for MediaWiki core the whole task takes less than 5 seconds. In the shared gate job with all of Wikibase+VE etc installed it takes 25 seconds in total. (Last two random jobs with these numbers: mediawiki-core-quibble#qunit, and wmf-quibble-gate#qunit).

Change 539630 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Dedicated Selenium job for gated repos

https://gerrit.wikimedia.org/r/539630

Change 539630 merged by jenkins-bot:
[integration/config@master] Dedicated Selenium job for gated repos

https://gerrit.wikimedia.org/r/539630

For the record, using cold caches:

wmf-quibble-selenium-hhvm-dockerSUCCESS15m 14s
wmf-quibble-selenium-php72-dockerSUCCESS10m 59s
hashar triaged this task as High priority.Sep 30 2019, 7:32 PM
hashar moved this task from Backlog to In progress on the Continuous-Integration-Config board.
Krinkle added a comment.EditedOct 1 2019, 4:53 AM

@hashar To confirm, we need only one of those, right? Afaik we don't need to cover server variants of DB or PHP for front-end tests.

Nice that php72 is faster. That makes the choice easier :D

Mentioned in SAL (#wikimedia-releng) [2019-10-02T03:07:31Z] <Krinkle> Fix "setup quibble mw-install" Jenkins Console section to not eat all output on new quibble-selenium jobs (T232759)

Krinkle closed this task as Resolved.Oct 3 2019, 1:53 AM
Krinkle assigned this task to hashar.
wmf-quibble-selenium-hhvm-dockerSUCCESS15m 14s
wmf-quibble-selenium-php72-dockerSUCCESS10m 59s

[..], we need only one of those, right? [..]

Now done, effectively, by 3791236e33.

hashar reopened this task as Open.EditedOct 4 2019, 10:16 AM

I have only extracted it for the wmf-quibble* jobs which are for core/extensions/skins gated together and just for master and wmf branches.

The regular quibble jobs would benefit from it as well. Eg for a Wikibase change we have:

JobDuration
mwgate-node10-docker4m 15s
quibble-vendor-mysql-php72-docker32m 09s (also run selenium tests)
quibble-vendor-mysql-php73-docker34m 56s (also run selenium tests)
wmf-quibble-vendor-mysql-php72-docker11m 06s (no more run selenium)
wmf-quibble-selenium-php72-docker12m 33s (split from above as a standalone job)
mwext-php72-phan-docker1m 02s
mwext-node10-rundoc-docker1m 51s
mwext-php72-phan-seccheck-docker2m 19s
mwselenium-quibble-docker3m 12s
wikibase-client-docker2m 23s
wikibase-repo-docker4m 30s

The reason I have not split Selenium tests out of quibble-vendor-mysql-php72-docker yet is that it is usually fast. Until it is triggered by a repository having lot of extensions with selenium tests. So probably we just have to split that for the biggest repositories, specially Wikibase which could most probably benefit from having dedicated / finely tuned jobs instead of the generic ones.

Yeah, the separation is most useful for the wmf-quibble job ("extension gate"). For others we would not be saving time as they are not the longest job in the pipeline.

Wikibase indeed could still benefit from it. Although I do wonder if the gate vs non-gate variants are useful for extensions that are in the gate. Especially given the number of dependencies Wikibase has, it might be just as good (or better) to not have the non-gate variant run there at all? Anyway, that might be a separate matter. Splitting seems useful either way. So the task remains open for splitting Wikibase's non-gate job?

I do wonder if the gate vs non-gate variants are useful for extensions that are in the gate. Especially given the number of dependencies Wikibase has, it might be just as good (or better) to not have the non-gate variant run there at all?

Yeah, that sounds sensible.

Change 543100 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Standalone Selenium jobs for mediawiki/core

https://gerrit.wikimedia.org/r/543100

Change 543101 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Skip selenium for mediawiki/core (run in standalone job)

https://gerrit.wikimedia.org/r/543101

Change 543100 merged by jenkins-bot:
[integration/config@master] Standalone Selenium jobs for mediawiki/core

https://gerrit.wikimedia.org/r/543100

Change 543101 merged by jenkins-bot:
[integration/config@master] Skip selenium for mediawiki/core (run in standalone job)

https://gerrit.wikimedia.org/r/543101

Change 545580 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Standalone Selenium jobs for extensions/skins

https://gerrit.wikimedia.org/r/545580

Change 545580 merged by jenkins-bot:
[integration/config@master] Standalone Selenium jobs for extensions/skins

https://gerrit.wikimedia.org/r/545580

Mentioned in SAL (#wikimedia-releng) [2019-10-23T17:00:53Z] <hashar> Reloading Zuul for https://gerrit.wikimedia.org/r/545580 Standalone Selenium jobs for Wikibase # T232759

For a Wikibase change on master:

JobDuration
quibble-vendor-mysql-php72-noselenium-docker15m 08s
quibble-vendor-selenium-docker12m 01s
wmf-quibble-vendor-mysql-php72-docker8m 44s
wmf-quibble-selenium-php72-docker13m 06s

That is better than almost half an hour!