Page MenuHomePhabricator

Switch mediawiki code coverage from xdebug to phpdbg or pcov
Open, Needs TriagePublic

Description

The generate of code coverage for mediawiki/core takes way too long (more than 3 hours) and eventually reach the 5 hours time limit (parent task T232706).

PHPUnit has support for phpdbg which is dramatically faster. An example for a single test:

xdebug

php tests/phpunit/phpunit.php tests//phpunit/includes/api/ApiQuerySiteinfoTest.php --filter testContinuation --coverage-html /tmp/mwcover-xdebug

Time: 45.86 seconds, Memory: 68.50MB

phpdbg (with Xdebug disabled)

phpdbg -qrr tests/phpunit/phpunit.php tests//phpunit/includes/api/ApiQuerySiteinfoTest.php --filter testContinuation --coverage-html /tmp/mwcover-phpdbg

Time: 7.62 seconds, Memory: 74.50MB

We should thus have CI to switch to phpdbg (and make sure Xdebug is disabled).

Some infos:

Event Timeline

hashar created this task.Sep 27 2019, 9:03 AM
hashar updated the task description. (Show Details)Sep 27 2019, 9:09 AM
hashar renamed this task from Switch mediawiki code coverage from xdebug to phpdbg to Switch mediawiki code coverage from xdebug to phpdbg or pcov.Sep 27 2019, 9:16 AM
hashar updated the task description. (Show Details)

I'd like to note that the latest version of xdebug (2.9.0), released one month ago, claims to be up to 250% faster than 2.7.0 (the version we're using) for generating coverage [1]. I've upgraded locally, and while I didn't get a 250% speedup, it's certainly faster (I didn't measure it, though). I believe this upgrade should be performed as soon as possible, as it seems fairly straightforward.

That said, phpdbg or pcov may still be faster. If we want to try the latter, then this task should have T192167 as subtask.

[1] - https://xdebug.org/announcements/2019-12-09

kostajh added subscribers: Gehel, kostajh.EditedJan 23 2020, 1:52 PM

While switching to pcov / phpdbg (pcov seems better after a quick look?) is a good idea, another consideration is to use only unit tests for generating coverage reports (cc @Gehel), which would speed things up by virtue of those tests running much faster and there also being fewer of them, for now.

But that would mean instead of recording 29% code coverage for core we would instead show 1.9%.

Now that we're on PHPUnit 8, I've fiddled a bit with pcov and, oh dear, that's just astonishing.

XDebug 2.9.1

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilterParser --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ 461 tests here ]

Time: 16.09 seconds, Memory: 70.00 MB

OK (461 tests, 1040 assertions)

Generating code coverage report in HTML format ... done [4.87 seconds]

Pcov

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilterParser --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ Same 461 tests here ]

Time: 3.88 seconds, Memory: 70.00 MB

OK (461 tests, 1040 assertions)

Generating code coverage report in HTML format ... done [2.1 seconds]

Multiple attempts result in similar times. As noted above, there are some differences in the reports. One that I could spot quickly is that PCOV considers the following to be not coverable:

  • lines containing only global $foo
  • lines containing only switch ( expr ) {

But I don't think it's a big deal.


Then I decided to execute more tests, including non-unit tests as well as database tests:

Xdebug
$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilter --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ 716 tests and 6 failures ]

Time: 4.19 minutes, Memory: 140.00 MB

FAILURES!
Tests: 716, Assertions: 1460, Failures: 6.

Generating code coverage report in HTML format ... done [4.95 seconds]
Pcov
$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilter --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ Still 716 tests ]

Time: 2.02 minutes, Memory: 178.00 MB

FAILURES!
Tests: 716, Assertions: 1460, Failures: 6.

Generating code coverage report in HTML format ... done [2.22 seconds]

Not too bad, huh?

Change 567938 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+

https://gerrit.wikimedia.org/r/567938

Daimona claimed this task.Jan 28 2020, 2:04 PM

As per T243847#6567940, I wanted to try xdebug 3.0.0 (released 5 days ago) and see what kind of speedup we can expect. The following measerements refer to running all tests in AbuseFilter.

Xdebug 2.9.1

Using PHP 7.4.1
[...939 tests]

Time: 4.51 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.96 seconds]

Xdebug 3.0.0, mode=coverage

Using PHP 7.4.1
[...939 tests]

Time: 3.07 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.12 seconds]

pcov

Using PHP 7.4.1
[...939 tests]

Time: 2.33 minutes, Memory: 176.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [3.87 seconds]

So yes, it does seem that xdebug brought a massive performance improvement, although it's still slower than pcov. I think that upgrading xdebug, if it can be done easily (i.e. without having to get pcov in apt or building from source), would be great.

Is pcov as accurate as xdebug? I remember that used to be a problem with phpdbg when I last looked into it a few years ago.

Is pcov as accurate as xdebug? I remember that used to be a problem with phpdbg when I last looked into it a few years ago.

It's very similar. According to my previous tests (T234020#5826901, which also match the documentation on github), the only notable difference is that pcov considers

  • lines containing only global $foo ...
  • lines containing only switch ( expr ) {

as not coverable. Which I think is not a problem, especially if it comes with that massive speedup.

As per T243847#6567940, I wanted to try xdebug 3.0.0 (released 5 days ago) and see what kind of speedup we can expect. The following measerements refer to running all tests in AbuseFilter.

Xdebug 2.9.1

Using PHP 7.4.1
[...939 tests]

Time: 4.51 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.96 seconds]

Xdebug 3.0.0, mode=coverage

Using PHP 7.4.1
[...939 tests]

Time: 3.07 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.12 seconds]

pcov

Using PHP 7.4.1
[...939 tests]

Time: 2.33 minutes, Memory: 176.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [3.87 seconds]

So yes, it does seem that xdebug brought a massive performance improvement, although it's still slower than pcov. I think that upgrading xdebug, if it can be done easily (i.e. without having to get pcov in apt or building from source), would be great.

@Daimona what is the command you used for this? Have you edited phpunit.xml to modify the <filter><whitelist> section for just AbuseFilter? It should be faster than that, I would think.

@Daimona what is the command you used for this? Have you edited phpunit.xml to modify the <filter><whitelist> section for just AbuseFilter? It should be faster than that, I would think.

Yes, I'm using a custom config file with only AbuseFilter whitelisted. The problem is that there are several heavy, slow, integration tests. Running without coverage, I get

Time: 1.85 minutes, Memory: 72.00 MB

Krinkle removed a subscriber: Krinkle.Dec 2 2020, 12:24 AM