| thcipriani | |
| Sep 11 2019, 4:16 PM |
| F31703336: Screenshot 2020-03-26 at 15.27.11.png | |
| Mar 26 2020, 3:28 PM |
| F31551572: jenkins-p95-mw-core (2).png | |
| Feb 6 2020, 12:42 AM |
| F30410393: jenkins-p95-mw-core (1).png | |
| Sep 20 2019, 8:55 PM |
| F30299642: jenkins-p95-mw-core.png | |
| Sep 11 2019, 4:16 PM |
Using the data collected by @dduvall in https://people.wikimedia.org/~dduvall/jenkins/ (using https://gerrit.wikimedia.org/g/releng/jenkins-build-stats/ ) I started graphing P-95 build-times for all projects (as an average), for mediawiki/* jobs (as an average), and for mediawiki/core (https://github.com/thcipriani/wikimedia-jenkins-stats/).
I noticed that there was an increase in the run time of jobs for mediawiki/core starting July 26th as seen in this graph:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T232626 CI mediawiki/core run times have increased since July 26th, 2019 | |||
| Resolved | awight | T231862 Selenium tests for Wikibase are being ran twice | |||
| Resolved | hashar | T232759 Move CI selenium/qunit tests of mediawiki repository to a standalone job | |||
| Resolved | hashar | T249845 CI jobs for WMF deployed extensions should use vendor.git , not composer | |||
| Resolved | • Mholloway | T237596 Add MachineVision dependencies to vendor | |||
| Declined | Reedy | T237588 Security review for MachineVision libraries |
I made a phabricator ticket at exact time: T228739: Jenkins runs Scribunto tests in Wikibase patches @Jdforrester-WMF
Another reason: T231862: Selenium tests for Wikibase are being ran twice (Wikibase is part of gate, lots of selenium tests are being ran twice)
I think that matches with the deployment of quibble 0.0.33/0.0.34. Notably b2f9e363046696ff1f547a9724a705bbd52dbde2 Run node browser tests in each repo (was T199116).
I would guess T231862 is the main offender indeed and @awight has a patch for Quibble (Wikibase tests are run once because that is the project being run and second time because Wikibase is listed as a dependency of WikibaseLexeme).
Quibble 0.0.35 (with selenium de-duping) has been released and is being used since 2019-09-17, but the p95 hasn't dropped significantly (still around 20–22 minutes), and certainly not back to ~12 minutes.
Do we have an idea of what we can next do to improve this?
From https://people.wikimedia.org/~dduvall/jenkins/2020-03-25/30-day/durations-by-project.html
Yep, we're back again around the 12-13 minute mark as was the status quo when this task was filed. I think that's all for this task? Boldly closing for now, but continuing with more ideas at T225730.
