Page MenuHomePhabricator

Some PHPUnit tests consistently fail when collecting coverage report
Closed, ResolvedPublic

Description

Last updated: 10 May 2018

https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3500/consoleText

  • There were 13 errors [..]
  • There were 9 failures [..]
There were 6 failures
1) ApiQueryContributionsTest::testSorting with data set "User IDs, write both" (1, array(array('ApiQueryContributionsTest A', 'ApiQueryContributionsTest B', 'ApiQueryContributionsTest C')), false, 9)
Batching with limit 1
Failed asserting that 10 is equal to 9 or is less than 9.

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/api/query/ApiQueryUserContributionsTest.php:115
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

2) ApiQueryContributionsTest::testSorting with data set "User IDs, write both, reverse" (1, array(array('ApiQueryContributionsTest A', 'ApiQueryContributionsTest B', 'ApiQueryContributionsTest C')), true, 9)
Result set is the same when batching with limit 1
Failed asserting that Array &0 (
    'ApiQueryContributionsTest B' => Array &1 (
        0 => 56
    )
) is identical to Array &0 (
    'ApiQueryContributionsTest B' => Array &1 (
        0 => 56
        1 => 63
        2 => 70
    )
    'ApiQueryContributionsTest A' => Array &2 (
        0 => 55
        1 => 62
        2 => 69
    )
    'ApiQueryContributionsTest C' => Array &3 (
        0 => 54
        1 => 61
        2 => 68
    )
).

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/api/query/ApiQueryUserContributionsTest.php:117
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

3) ApiQueryContributionsTest::testSorting with data set "User IDs, write new" (2, array(array('ApiQueryContributionsTest A', 'ApiQueryContributionsTest B', 'ApiQueryContributionsTest C')), false, 9)
Batching with limit 1
Failed asserting that 10 is equal to 9 or is less than 9.

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/api/query/ApiQueryUserContributionsTest.php:115
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

4) ApiQueryContributionsTest::testSorting with data set "User IDs, write new, reverse" (2, array(array('ApiQueryContributionsTest A', 'ApiQueryContributionsTest B', 'ApiQueryContributionsTest C')), true, 9)
Result set is the same when batching with limit 1
Failed asserting that Array &0 (
    'ApiQueryContributionsTest B' => Array &1 (
        0 => 56
    )
) is identical to Array &0 (
    'ApiQueryContributionsTest B' => Array &1 (
        0 => 56
        1 => 63
        2 => 70
    )
    'ApiQueryContributionsTest A' => Array &2 (
        0 => 55
        1 => 62
        2 => 69
    )
    'ApiQueryContributionsTest C' => Array &3 (
        0 => 54
        1 => 61
        2 => 68
    )
).

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/api/query/ApiQueryUserContributionsTest.php:117
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

5) WikiPageContentHandlerDbTest::testUpdateRedirectOn with data set #3 ('SomeText', false, 'Foo', false, false, 1)
Success assertion
Failed asserting that true matches expected false.

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/page/WikiPageDbTestBase.php:1361
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

6) WikiPageNoContentHandlerDbTest::testUpdateRedirectOn with data set #3 ('SomeText', false, 'Foo', false, false, 1)
Success assertion
Failed asserting that true matches expected false.

/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/includes/page/WikiPageDbTestBase.php:1361
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/tests/phpunit/MediaWikiTestCase.php:423
/srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src/maintenance/doMaintenance.php:94

Details

Related Gerrit Patches:

Event Timeline

Paladox created this task.Dec 29 2017, 12:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 29 2017, 12:50 PM

Hmmm, what is the difference in the setup for tests / phpunit between the regular mediawiki phpunit test jobs and the coverage job?

SQLite instead of MySQL, PHP 5.6 instead of PHP 5.5/HHVM, runs on a permanent slave instead of a nodepool instance, ...

Change 406604 had a related patch set uploaded (by Krinkle; owner: Addshore):
[mediawiki/core@master] Add tablesUsed to RevisionStoreDbTest

https://gerrit.wikimedia.org/r/406604

The job always passes whether tests fail or not. If you look at https://integration.wikimedia.org/ci/view/Default/job/mediawiki-core-code-coverage/3300/console you can still see the tests failing.

Change 406604 merged by jenkins-bot:
[mediawiki/core@master] Add tablesUsed to RevisionStoreDbTest

https://gerrit.wikimedia.org/r/406604

Krinkle updated the task description. (Show Details)Mar 2 2018, 2:12 AM
Krinkle added a subscriber: Krinkle.

Updated.

From 4 errors and 4 failures in December 2017... now up to 13 errors and 9 failures in March 2018. Yikes!

Krinkle renamed this task from mediawiki tests fail in mediawiki code coverage test to Some PHPUnit tests consistently fail when collecting coverage report.May 10 2018, 6:08 PM
Krinkle updated the task description. (Show Details)
hashar closed this task as Resolved.Apr 4 2019, 8:46 PM
hashar added a subscriber: hashar.

Indeed, it seems like the build is clean nowadays:

01:18:43.459 OK, but incomplete, skipped, or risky tests!
01:18:43.459 Tests: 20837, Assertions: 163050, Skipped: 116.