For the "typical" time it takes for a commit to be approved and landed in master to be 5 minutes or less.
As of 11 June 2019, the gate usually takes around 20 minutes.
The two slowest jobs typically take 13-17 minutes each. The time for the gate overall is rarely under 15 minutes, because we run multiple of these jobs (increasing the chances of random slowness), and while they can run in parallel, they don't always start immediately - given limited CI execution slots.
Below a sample from a MediaWiki commit (master branch):
These can be grouped in two kinds of jobs:
- wmf-quibble: These install MW with the gated extensions, and then run all PHPUnit, Selenium and QUnit tests.
- mediawiki-quibble: These install MW bundled extensions only, and then run PHPUnit, Selenium and QUnit tests.
Stats from wmf-quibble-core-vendor-mysql-php72-docker
- 9-15 minutes (mediawiki-core, master, gate-and-submit)
- PHPUnit (dbless): 1.91 minutes / 15,782 tests.
- QUnit: 29 seconds / 1286 tests.
- Selenium: 143 seconds / 43 tests.
- PHPUnit (db): 3.85 minutes / 4377 tests.
Stats from wmf-quibble-core-vendor-mysql-hhvm-docker:
- 13-17 minutes (mediawiki-core, master, gate-and-submit)
- PHPUnit (dbless): 2.43 minutes / 15,782 tests.
- QUnit: 32 seconds / 1286 tests.
- Selenium: 187 seconds / 43 tests.
- PHPUnit (db): 4.78 minutes / 4377 tests.
Scope of task
This task represents the goal of reaching 5 minutes or less. The work tracked here includes researching ways to get there, trying them out, and putting one or more ideas into practice. The task can be closed once we have reached the goal or if we have concluded it isn't feasible or useful.
Feel free to add/remove subtasks as we go along and consider different things.
- T202116: LoadBalancer opening extra connections in different connection categories doesn't work with PHPUnit & temporary tables
- T225496: Improve caching in CI tests
- T225901: Don't deduplicate archive table on new installs
- T225719: HashRingTest::testHashRingKetamaMode takes 3+ seconds
- T225184: CirrusSearch\SearcherTest::testSearchText PHPUnit tests take a while and runs for everyone
- T227067: ReleaseNotesTest:testReleaseNotesFilesExistAndAreNotMalformed takes ~ 4 seconds
- T196347: Quibble may need to rebuild localization cache before running tests
- Speed up LocalisationCache in Quibble jobs by switching from DB to file-based (by setting $wgCacheDirectory). – T225730#5400979
- Skip "npm test" step during Quibble jobs (covered by separate node10-test job). T233143
Ideas to explore and related work
- Look at the PHPUnit "Test Report" for a commit and sort the root by duration. Find the slowest ones and look at its test suite to look for ways to improve it. Is it repeating expensive setups? Perhaps that can be skipped or re-used. Is it running hundreds of variations for the same integration test? Perhaps reduce it to just 1 case for that story, and apply the remaining cases to a lighter unit test instead.
- T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel
- T221434: Ensure we're testing appropriately and not over-testing across Wikimedia-deployed code
- T222757: quibble-vendor-mysql-hhvm-docker for WikibaseCirrusSearch takes over 40 minutes
- T225068: Add a PHPUnit group to skip test on gated CI runs
- T225218: Consider httpd for quibble instead of php built-in server
- T225248: Consider moving browser based tests (Selenium and QUnit) to a non-voting pipeline
- T230701: Migrate Scribunto to stop using MediaWikiIntegrationTestCase on unit tests