Page MenuHomePhabricator

EventBus jobs failing heavily because of CentralNotice and WikibaseRepo
Open, Needs TriagePublic

Description

In the Event-Platform extension (for example in patch https://gerrit.wikimedia.org/r/514518), this currently happens:

18:08:03 INFO:test.run_phpunit:php tests/phpunit/phpunit.php --debug-tests --testsuite extensions --exclude-group Broken,ParserFuzz,Stub,Database --log-junit /workspace/log/junit-dbless.xml
…
18:17:56 OK, but incomplete, skipped, or risky tests!
18:17:56 Tests: 23303, Assertions: 165192, Skipped: 117.
…
18:29:10 INFO:test.run_phpunit:php tests/phpunit/phpunit.php --debug-tests --testsuite extensions --group Database --exclude-group Broken,ParserFuzz,Stub --log-junit /workspace/log/junit-db.xml
…
18:38:48 FAILURES!
18:38:48 Tests: 5441, Assertions: 16843, Errors: 29, Failures: 89, Skipped: 85.

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52053/consoleFull

I counted, and 97 of these errors and failures are from MediaWiki-extensions-CentralNotice, and the remaining 21 from Wikibase.

How to report this? What is even the issue? What makes me wonder is the fact that the job runs PHPUnit two times, and the first time everything succeeds. The majority of the Wikibase tests seems to fail because of a TokenCheckException. I have no idea what makes CentralNotice fail. A polluted ObjectCache maybe?

Details

Related Gerrit Patches:
mediawiki/extensions/EventBus : masterFix CentralNoticeCampaignChange hook handler failing with null's
integration/config : masterRevert "[EventBus] Add dependency"
mediawiki/extensions/CentralNotice : masterFix CentralNoticeCampaignChange hook not passign expected arrays

Event Timeline

Restricted Application added projects: Wikidata, Analytics. · View Herald TranscriptJun 6 2019, 10:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

the job runs PHPUnit two times, and the first time everything succeeds

This just means that the failing tests are all in the Database group. These are skipped in the first run, and included in the second.

Looking at the first set of failures coming from CentralNotice, there's a two-way dependency between EventBus and CentralNotice, and it seems to be the only extension wired this way. But the issue seems to be Campaign::processAfterCampaignChange, which is called with beginSettings and endSettings = []. This always returns null from processSettingsForHook, which then crashes when an array is expected in EventFactory. I don't understand how this worked until now.

The CentralNotice changes I'm looking at were made in February, fac8406c28d9, and the EventBugHooks side was changed May 17 in b0450e28449c. No clue why the tests wouldn't have failed until now.

Change 514701 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/EventBus@master] Fix CentralNoticeCampaignChange hook handler failing with null's

https://gerrit.wikimedia.org/r/514701

Change 514703 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/CentralNotice@master] Fix CentralNoticeCampaignChange hook not passign expected arrays

https://gerrit.wikimedia.org/r/514703

So I suppose since those patches are in two different repos, tests on each fail until they are both merged?

I think we should merge them. This is blocking other EventBus patches too.

awight added a comment.Jun 6 2019, 1:58 PM

So I suppose since those patches are in two different repos, tests on each fail until they are both merged?

They are redundant rather than interdependent, either patch should have fixed the issue...

The failures I see in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52212/console don't really have to do with nulls passed to EventBus though. They just look failing CentralNotice tests, possible due to bad fixture data? E.g.

...
-        'throttle' => 50
+        'throttle' => 100
...

in CentralNotice/tests/phpunit/CentralNoticeTestFixtures.php

Change 514703 abandoned by Thiemo Kreuz (WMDE):
Fix CentralNoticeCampaignChange hook not passign expected arrays

https://gerrit.wikimedia.org/r/514703

fdans moved this task from Incoming to Radar on the Analytics board.Jun 6 2019, 4:44 PM

@Umherirrender I'd like to revert https://gerrit.wikimedia.org/r/c/integration/config/+/513663 as it is causing blockers for EventBus changes. Objections?

I am fine with revert of dependency

Change 514841 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[integration/config@master] Revert "[EventBus] Add dependency"

https://gerrit.wikimedia.org/r/514841

Change 514841 merged by jenkins-bot:
[integration/config@master] Revert "[EventBus] Add dependency"

https://gerrit.wikimedia.org/r/514841

Mentioned in SAL (#wikimedia-releng) [2019-06-06T18:06:32Z] <James_F> Reloading Zuul to drop EventBus dependency on CentralNotice for now T225195

Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:25 PM

Can we just declare this Resolved or is further work under this banner planned?

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM

Change 514701 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Fix CentralNoticeCampaignChange hook handler failing with null's

https://gerrit.wikimedia.org/r/514701