Page MenuHomePhabricator
Create Task
Maniphest T70653

Fix flaky unit test "TextPassDumperTest::testCheckpointGzip"
Open, MediumPublic
Actions

Description

The computer in question was definitely not too fast, what's actually going on?

  1. TextPassDumperTest::testCheckpointPlain

expected more than 1 checkpoint to have been created. Checkpoint interval is 0.5 seconds, maybe your computer is too fast?
Failed asserting that 1 is greater than 1.

/srv/vagrant/mediawiki/tests/phpunit/maintenance/backupTextPassTest.php:397
/srv/vagrant/mediawiki/tests/phpunit/maintenance/backupTextPassTest.php:406
/srv/vagrant/mediawiki/tests/phpunit/MediaWikiTestCase.php:133
/srv/vagrant/mediawiki/tests/phpunit/MediaWikiPHPUnitCommand.php:42
/srv/vagrant/mediawiki/tests/phpunit/phpunit.php:160

  1. TextPassDumperTest::testCheckpointGzip

expected more than 1 checkpoint to have been created. Checkpoint interval is 0.5 seconds, maybe your computer is too fast?
Failed asserting that 1 is greater than 1.

/srv/vagrant/mediawiki/tests/phpunit/maintenance/backupTextPassTest.php:397
/srv/vagrant/mediawiki/tests/phpunit/maintenance/backupTextPassTest.php:423
/srv/vagrant/mediawiki/tests/phpunit/MediaWikiTestCase.php:133
/srv/vagrant/mediawiki/tests/phpunit/MediaWikiPHPUnitCommand.php:42
/srv/vagrant/mediawiki/tests/phpunit/phpunit.php:160

Also seen at https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/37/console

Version: unspecified
Severity: normal

Details

Reference
bz68653
SubjectRepoBranchLines +/-
backupTextPassTest: Disable testCheckpointGzip testmediawiki/coremaster+3 -0
Fix and re-enable Dumps' checkpoint testsmediawiki/coremaster+4 -7
Allow to set stub read buffer size for TextPassDumpermediawiki/coremaster+69 -4
backupTextPassTest: Disable checkpointHelper testmediawiki/coremaster+6 -0
Customize query in gerrit

Revisions and Commits

rMW MediaWiki
rMWa3f19ee08269 Disable flaky test 'testCheckpointPlain'
rMW635bf3f48f8f backupTextPassTest: Disable flaky test 'testCheckpointPlain'
rMWe3266ae87e18 backupTextPassTest: Disable flaky test 'testCheckpointPlain'

Related Objects

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 22 2014, 3:39 AM
• bzimport added a project: MediaWiki-Core-Tests.
• bzimport set Reference to bz68653.
• bzimport added a subscriber: Unknown Object (MLST).
Nemo_bis created this task.Jul 27 2014, 12:43 AM
hashar added a comment.Jul 28 2014, 10:05 AM

Seems to be a standalone issue, hence not blocking Bug 67216 - Have unit tests of all wmf deployed extensions pass when installed together.

Nemo_bis updated the task description. (Show Details)Dec 15 2014, 11:37 PM
Nemo_bis set Security to None.
Nemo_bis added a subscriber: awight.
Anomie subscribed.Jan 9 2015, 6:15 PM

This is causing bogus test failures in Jenkins now, see https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/1008/console for example.

Krinkle raised the priority of this task from Low to High.Feb 12 2015, 8:55 AM
Krinkle added a project: Continuous-Integration-Infrastructure.

Now happening on mediawiki-core jobs, too.

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/2566/console

gerritbot added a project: Patch-For-Review.Feb 12 2015, 8:57 AM
gerritbot subscribed.

Change 190173 had a related patch set uploaded (by Krinkle):
backupTextPassTest: Disable checkpointHelper test

https://gerrit.wikimedia.org/r/190173

Patch-For-Review

gerritbot added a comment.Feb 13 2015, 12:48 AM

Change 190173 merged by jenkins-bot:
backupTextPassTest: Disable checkpointHelper test

https://gerrit.wikimedia.org/r/190173

Krinkle mentioned this in rMW3d4c0e933f41: backupTextPassTest: Disable checkpointHelper test.Feb 13 2015, 12:48 AM
Krinkle renamed this task from TextPassDumperTest::testCheckpointGzip expected more than 1 checkpoint to have been created to Fix flaky unit test "TextPassDumperTest::testCheckpointGzip".Feb 13 2015, 2:12 AM
Krinkle lowered the priority of this task from High to Medium.
Krinkle removed projects: Patch-For-Review, Continuous-Integration-Infrastructure.
Krinkle added a project: Continuous-Integration-Infrastructure.
Krinkle removed subscribers: Krinkle, Unknown Object (MLST).
Krinkle subscribed.

Test has been disabled. The Continuous-Integration-Infrastructure blocked has been solved. Retriaged as MediaWiki core issue to fix this unit test.

Krinkle removed a project: Continuous-Integration-Infrastructure.Feb 13 2015, 2:13 AM
Krinkle unsubscribed.
Nemo_bis assigned this task to QChris.Feb 13 2015, 8:26 AM

Reassigning per QChris:

I did not even remember I did :-D I'll only find time to look at it over the weekend.

Please unassign if you don't have time for it. :)

hashar added a subscriber: Krinkle.Feb 13 2015, 10:11 AM

Thanks @Krinkle and @QChris , I was rather conservative turns out the test has been failing quite often so indeed it is safer to disable for now.

gerritbot added a project: Patch-For-Review.Feb 16 2015, 11:30 PM

Change 190953 had a related patch set uploaded (by QChris):
Fix and re-enable Dumps' checkpoint tests

https://gerrit.wikimedia.org/r/190953

Patch-For-Review

gerritbot added a comment.Feb 19 2015, 11:50 PM

Change 191814 had a related patch set uploaded (by QChris):
Allow to set stub read buffer size for TextPassDumper

https://gerrit.wikimedia.org/r/191814

Patch-For-Review

gerritbot added a comment.Mar 3 2015, 9:35 PM

Change 191814 merged by jenkins-bot:
Allow to set stub read buffer size for TextPassDumper

https://gerrit.wikimedia.org/r/191814

gerritbot added a comment.Mar 3 2015, 9:35 PM

Change 190953 merged by jenkins-bot:
Fix and re-enable Dumps' checkpoint tests

https://gerrit.wikimedia.org/r/190953

QChris mentioned this in rMW7f687cc858ed: Allow to set stub read buffer size for TextPassDumper.Mar 3 2015, 9:35 PM
QChris mentioned this in rMW73c6f525e0fa: Fix and re-enable Dumps' checkpoint tests.
hashar closed this task as Resolved.Mar 3 2015, 9:39 PM

Thank you @QChris !

Krinkle reopened this task as Open.Apr 16 2015, 2:59 AM

Failing again.

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/4787/console
For unrelated patch set https://gerrit.wikimedia.org/r/#/c/204426/

02:55:39 There was 1 failure:
02:55:39 
02:55:39 1) TextPassDumperDatabaseTest::testCheckpointGzip
02:55:39 Skipping past end of siteinfo
02:55:39 Failed asserting that false is true.
02:55:39 
02:55:39 /mnt/jenkins-workspace/workspace/mediawiki-phpunit-zend/src/tests/phpunit/maintenance/DumpTestCase.php:178
02:55:39 /mnt/jenkins-workspace/workspace/mediawiki-phpunit-zend/src/tests/phpunit/maintenance/backupTextPassTest.php:331
02:55:39 /mnt/jenkins-workspace/workspace/mediawiki-phpunit-zend/src/tests/phpunit/maintenance/backupTextPassTest.php:441
02:55:39 /mnt/jenkins-workspace/workspace/mediawiki-phpunit-zend/src/tests/phpunit/MediaWikiTestCase.php:131
gerritbot added a comment.Apr 16 2015, 3:04 AM

Change 204447 had a related patch set uploaded (by Krinkle):
backupTextPassTest: Disable testCheckpointGzip test

https://gerrit.wikimedia.org/r/204447

gerritbot added a comment.Apr 16 2015, 3:30 AM

Change 204447 merged by jenkins-bot:
backupTextPassTest: Disable testCheckpointGzip test

https://gerrit.wikimedia.org/r/204447

Krinkle mentioned this in rMW9107fb539111: backupTextPassTest: Disable testCheckpointGzip test.Apr 16 2015, 3:30 AM
QChris removed QChris as the assignee of this task.Apr 16 2015, 11:07 PM
QChris subscribed.
In T70653#1211336, @Krinkle wrote:

Failing again.

The Skipping past end of siteinfo log message means that no </siteinfo> could get found in the dump file.

Did this happen more often recently or was this just a one-off fluke?

(I tried to reproduce locally, but couldn't. Tried to reproduce through WMF's Jenkins in https://gerrit.wikimedia.org/r/#/c/204645/ but couldn't either)

Krinkle added a comment.Apr 16 2015, 11:38 PM

I saw it happen on two commits in one day. Two isn't much, but I'm applying zero tolerance. A few sources of flakiness recently: MediaWiki core, HHVM, Wikimedia Labs networking and disk I/O, Jenkins deadlocks, Zuul deadlocks, Gerrit event stream, npm/composer cache corruption, Xvfb conflicts, corrupt DNS resolution. This makes for an unstable platform that is impossible to maintain (for CI admins) and unpleasant to work with (for MediaWiki developers and SWAT deployers).

I am happy to help put in place additional debugging for when it fails again. Race conditions are hard. But it's only one of dozens of race conditions we dealt with lately. I'm working on making the wider platform stable as we're yet to have a 24 hour period without CI interruption of some sort (since early 2014).

Krinkle added a comment.Jul 7 2015, 6:41 PM

Here's another one:

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/10886/consoleFull

1) TextPassDumperDatabaseTest::testCheckpointPlain
Skipping past end of siteinfo
Failed asserting that false is true.
Krinkle added a commit: rMWe3266ae87e18: backupTextPassTest: Disable flaky test 'testCheckpointPlain'.Jul 7 2015, 6:54 PM
Krinkle added a comment.Jul 7 2015, 9:14 PM

And again, on a wmf branch.
https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/6367/console
Backporting disablement for now.

Krinkle added a commit: rMW635bf3f48f8f: backupTextPassTest: Disable flaky test 'testCheckpointPlain'.Jul 7 2015, 9:23 PM
Krinkle added a commit: rMWa3f19ee08269: Disable flaky test 'testCheckpointPlain'.Apr 28 2016, 3:08 AM
Krinkle edited projects, added MediaWiki-Core-Snapshots; removed Patch-For-Review.Jan 13 2017, 6:04 AM
Krinkle added a subscriber: ArielGlenn.
ArielGlenn added a comment.Jan 16 2017, 8:50 PM

I'm going to ping @QChris again since he's around and active.

Krinkle unsubscribed.Feb 25 2017, 4:30 AM
Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Tests board.Jul 7 2017, 5:13 AM
Krinkle added a project: Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Mar 20 2018, 2:32 AM
Krinkle moved this task from Untriaged to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.
awight unsubscribed.Mar 21 2019, 4:07 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:12 PM
Krinkle edited projects, added ci-test-error (WMF-deployed Build Failure); removed Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Apr 18 2020, 10:54 PM
Krinkle removed a project: ci-test-error (WMF-deployed Build Failure).Apr 19 2020, 5:56 PM
Krinkle changed the subtype of this task from "Production Error" to "Task".
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL