Page MenuHomePhabricator

Speed up mediawiki-phpunit-hhvm and mediawiki-phpunit-zend Jenkins jobs (parsertests are slow)
Closed, ResolvedPublic

Description

The MediaWiki test suite is painfully slow and delays merging of changes.

A test report for the last successful build is available via: https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/lastSuccessfulBuild/testReport/(root)/

What it shows is more than half of the time is spent in ParserTest_Parser⁄parserTests::testParserTest which is the PHPUnit wrapping class for the parser tests.

I believe it is terrible either because the NewParserTests class is horrible or because of all the PHPUnit overhead.

The good old tests/parserTests.php that were the first tests we ever had, is way faster. Less than a minute on my laptop.

So potentially, we could filter out the PHPUnit based class, or even get rid of it since lot of its code has been copy pasted from tests/parserTests.php. To integrate it in Jenkins we would run the usual PHPUnit suite then the old tests/parserTests.php.

This way if other tests are falling, we will not even run the parser tests.

We probably want to test tests/parserTests.php to output JUnit format so it can be recorded by Jenkins. But it is not strictly necessary --quiet suppress success and gives good enough output:

Event Timeline

hashar created this task.Sep 30 2015, 9:31 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2015, 9:31 PM
hashar triaged this task as High priority.Oct 6 2015, 11:25 AM
hashar set Security to None.
ssastry moved this task from Backlog to Non-Parsoid Tasks on the Parsoid board.Oct 19 2015, 11:12 PM

From various discussions I had since I filled this task, it seems people appreciate the PHUnit wrapper. To speed up the Jenkins job, we could just isolate the parser tests and exclude them from the job. Then add a new job that runs solely the parser tests.

By running both corpus in parallel, we would speed up the time to report back to developers / gate processing (i.e.: time for CI to merge).

hashar claimed this task.Nov 30 2015, 11:38 AM

Change 255569 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/255569

Change 255992 had a related patch set uploaded (by Hashar):
Env configurable wrapper for MW phpunit.php

https://gerrit.wikimedia.org/r/255992

Change 255992 merged by jenkins-bot:
Env configurable wrapper for MW phpunit.php

https://gerrit.wikimedia.org/r/255992

Change 256006 had a related patch set uploaded (by Hashar):
Extract mw parser tests to standalone jobs

https://gerrit.wikimedia.org/r/256006

Change 255569 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/255569

Change 256215 had a related patch set uploaded (by Hashar):
Replace mw-run-phpunit.sh with mw-phpunit.sh

https://gerrit.wikimedia.org/r/256215

Change 256215 merged by jenkins-bot:
Replace mw-run-phpunit.sh with mw-phpunit.sh

https://gerrit.wikimedia.org/r/256215

Change 256006 merged by jenkins-bot:
Extract mw parser tests to standalone jobs

https://gerrit.wikimedia.org/r/256006

Change 256233 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256233

Change 256234 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256234

Change 256249 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256249

Change 256250 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256250

Change 256252 had a related patch set uploaded (by Hashar):
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256252

hashar added a comment.Dec 1 2015, 5:06 PM

The MediaWiki PHPUnit test suite is run for Zend with two jobs from now on for all branches we support. The new job is mediawiki-phpunit-parsertests-zend.

hashar renamed this task from Speed up mediawiki-phpunit-hhvm and mediawiki-phpunit-zend Jenkins jobs to Speed up mediawiki-phpunit-hhvm and mediawiki-phpunit-zend Jenkins jobs (parsertests are slow).Dec 1 2015, 5:07 PM

Change 256233 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256233

hashar closed this task as Resolved.Dec 1 2015, 5:13 PM

Splitting the parsertests pretty much implements the purpose of this task: find a way to speed up parser tests run. They are still slow, but at least run in parallel now for Zend.

The HHVM flavor has not been split, the job runs the whole suite fast enough (less than 4 minutes all included).

Change 256234 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256234

Change 256249 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256249

Change 256250 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256250

Change 256252 merged by jenkins-bot:
tests: let us select/exclude ParserTests

https://gerrit.wikimedia.org/r/256252

hashar added a comment.Dec 1 2015, 9:52 PM

Poked wikitech-l about it:

CI parsertests now run standalone for Zend
https://lists.wikimedia.org/pipermail/wikitech-l/2015-December/084181.html