## Context
Our CI runs are taking approx. 25 minutes, which substantially slows down our development - review - integration cycle. This causes more context switching, feedback cycles are longer. This also reduces the amount of patches we can backport in a deployment window.
There were previous attempts to improve this situation, both generally and for Wikibase (and related extensions) in particular. However, they have seen a slow amount of progress in the last 5 years, but the length of CI jobs is trending upwards. These tasks are:
- {T225730}
- {T50217}
- {T226869}
## Main Objectives
In the scope of this epic we would like to:
- Speed up CI to under 10 minutes, and revise the goal as soon as we reach it, for posterity and additional work.
- Monitor our CI Speeds so that we can act upon future slowdowns.
## Mitigated Risks
If we speed up CI:
- We could potentially backport more changes
- Increase development - release cycle efficiency
- Remove disincentives to add tests in fear of slowing down CI even further
## Potential Tasks
...
## History
{T225730} sets an ambitious goal for CI job runs, and documents the progress that has been made towards the goal. Since the task was filed in 2019, many steps have been undertaken to optimise the builds including, but not limited to:
- Improving CI caching and generally reducing CI job setup time
- Reducing the runtime of specific long-running tests
- Skipping some tests on gated CI runs
- Fixing bugs that caused unnecessary work during CI
Work has also been started on:
- {T50217}
- {T226869}
There is a specific ticket about a slowdown that occurred in 2019 for Wikibase tests: {T228739}. Supposedly the runtime of the tests doubled on July 22nd 2019.
Running the tests today (27. March 2024), the timings break down as follows:
```lines=10
Analysis for change mediawiki/extensions/Wikibase~1015066:
quibble-vendor-mysql-php74-noselenium-docker (165337): 360
Setup: 8
Versions: 1
Ensure dir: '/workspace/log': 0
Zuul clone : 83
Submodule update: /workspace/src: 13
Run npm and composer tests: 0
composer test in /workspace/src/extensions/Wikibase: 28
Revert to git clean -xqdf in /workspace/src/extensions/Wikibase: 0
Install composer dev-requires for vendor.git: 11
Start backends: <MySQL (no socket)>: 4
Run Post-dependency install: 0
Install MediaWiki: 8
npm install in /workspace/src: 1
PHPUnit unit tests: 13
PHPUnit extensions suite (without database or standalone): 181
PostBuildScript: 9
quibble-vendor-mysql-php74-selenium-docker (102918): 775
Setup: 54
Versions: 1
Ensure dir: '/workspace/log': 0
Zuul clone : 81
Submodule update: /workspace/src: 13
Install composer dev-requires for vendor.git: 10
Start backends: <MySQL (no socket)>: 3
Run Post-dependency install: 0
Install MediaWiki: 10
npm install in /workspace/src: 1
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
Browser tests: mediawiki/extensions/Wikibase: 0
Selenium extensions/Wikibase: 236
Selenium extensions/AbuseFilter: 36
Selenium extensions/CheckUser: 26
Selenium extensions/Cite: 114
Selenium extensions/Echo: 16
Selenium extensions/Popups: 47
Selenium extensions/VisualEditor: 39
Selenium skins/MinervaNeue: 26
PostBuildScript: 62
mwgate-node18-docker (32470): 297
Setup: 248
PostBuildScript: 49
mwext-php74-phan-docker (109326): 108
Setup: 7
Versions: 0
Ensure dir: '/workspace/log': 0
Zuul clone : 33
Submodule update: /workspace/src: 2
Run Post-dependency install: 66
wmf-quibble-vendor-mysql-php74-docker (56161): 1294
Setup: 8
Versions: 0
Ensure dir: '/workspace/log': 0
Zuul clone : 116
Submodule update: /workspace/src: 8
Run npm and composer tests: 0
composer test in /workspace/src/extensions/Wikibase: 24
Revert to git clean -xqdf in /workspace/src/extensions/Wikibase: 1
Install composer dev-requires for vendor.git: 9
Start backends: <MySQL (no socket)>: 4
Run Post-dependency install: 0
Install MediaWiki: 4
npm install in /workspace/src: 3
PHPUnit unit tests: 14
PHPUnit extensions suite (without database or standalone): 157
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
Run Qunit tests: 33
PHPUnit extensions suite (with database): 872
PostBuildScript: 41
wmf-quibble-selenium-php74-docker (90455): 804
Setup: 17
Versions: 1
Ensure dir: '/workspace/log': 0
Zuul clone : 95
Submodule update: /workspace/src: 9
Install composer dev-requires for vendor.git: 9
Start backends: <MySQL (no socket)>: 4
Run Post-dependency install: 0
Install MediaWiki: 3
npm install in /workspace/src: 6
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
Browser tests: mediawiki/extensions/Wikibase: 0
Selenium extensions/Wikibase: 225
Selenium extensions/AbuseFilter: 36
Selenium extensions/CheckUser: 26
Selenium extensions/Cite: 43
Selenium extensions/Echo: 14
Selenium extensions/FileImporter: 17
Selenium extensions/GrowthExperiments: 75
Selenium extensions/Math: 12
Selenium extensions/PageTriage: 51
Selenium extensions/ProofreadPage: 31
Selenium extensions/VisualEditor: 40
Selenium skins/MinervaNeue: 26
PostBuildScript: 64
mwext-node18-rundoc-docker (4149): 129
Setup: 116
PostBuildScript: 13
mediawiki-quibble-apitests-vendor-php74-docker (61067): 697
Setup: 42
Versions: 1
Ensure dir: '/workspace/log': 0
Zuul clone : 57
Submodule update: /workspace/src: 12
Install composer dev-requires for vendor.git: 8
Start backends: <MySQL (no socket)>: 4
Run Post-dependency install: 0
Install MediaWiki: 5
npm install in /workspace/src: 1
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
Run API-Testing: 544
PostBuildScript: 23
wikibase-client-docker (38572): 100
Setup: 8
Versions: 0
Ensure dir: '/workspace/log': 0
Zuul clone : 22
Submodule update: /workspace/src: 3
Create composer.local.json with dependencies ['mediawiki/core': 0
composer update for mediawiki/core: 11
Start backends: <MySQL (no socket)>: 4
Run Post-dependency install: 0
Install MediaWiki: 3
npm install in /workspace/src: 4
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
User commands: extensions/Wikibase/build/jenkins/mw-apply-wb-settings.sh -r client: 38
PostBuildScript: 7
wikibase-repo-docker (38571): 199
Setup: 7
Versions: 1
Ensure dir: '/workspace/log': 0
Zuul clone : 24
Submodule update: /workspace/src: 3
Create composer.local.json with dependencies ['mediawiki/core': 0
composer update for mediawiki/core: 12
Start backends: <quibble.backend.SQLite object at 0x7faf66399b00>: 0
Run Post-dependency install: 0
Install MediaWiki: 3
npm install in /workspace/src: 6
Start backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>: 0
User commands: extensions/Wikibase/build/jenkins/mw-apply-wb-settings.sh -r repo: 133
PostBuildScript: 10
```
(analysis courtesy of [[https://gitlab.wikimedia.org/arthurtaylor/jenkins-run-analysis|jenkins-run-analysis]])
This places `wmf-quibble-vendor-mysql-php74-docker` on the critical path with a runtime of 1294 seconds (21.6 minutes) with the longest time being spent in `PHPUnit extensions suite (with database)` - 872 seconds (14.5 minutes). Any optimisation here would directly reduce the total time before a result comes back from CI.