Page MenuHomePhabricator
Create Task
Maniphest T237551

beta wikidata trying to dispatch to wikis that don't exist
Closed, ResolvedPublic
Actions

Description

https://wikidata.beta.wmflabs.org/wiki/Special:DispatchStats

Dispatch statistics

SitePositionPendingLagTouched
Freshestaawiki167647100 minutes17:15, 6 November 2019
Mediancswikiversity166052315,94890 days, 8 hours and 4 minutes11:49, 7 August 2019
Stalestndswikiquote166052315,94890 days, 8 hours and 4 minutes11:50, 7 August 2019
Average--15,01285 days and 56 minutes-

But the median and stalest wikis don't exist in beta... So it shouldn't dispatch to them

As the "freshest" does exist, this might not be causing any actual problems. It just looks odd and presumably spamming the logs

reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki
17:15:48 Starting loop for unlimited passes or 540 seconds
17:15:51 ERROR: LogicException: Domain 'afwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('afwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
^C
reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki
17:16:06 Starting loop for unlimited passes or 540 seconds
17:16:06 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:21 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:33 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:44 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Failed to grab dispatch lock for afwiki
17:16:49 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:02 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:23 ERROR: LogicException: Domain 'azwikiquote' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('azwikiquote', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:37 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:48 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:03 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:14 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:26 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:37 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:49 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:19:00 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
^C

Related Objects

Event Timeline

Reedy created this task.Nov 6 2019, 5:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2019, 5:25 PM
ArielGlenn subscribed.Nov 6 2019, 5:26 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Nov 6 2019, 5:28 PM
Lucas_Werkmeister_WMDE subscribed.
Addshore triaged this task as Medium priority.Nov 16 2019, 9:15 PM
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
WMDE-leszek added subscribers: Addshore, WMDE-leszek.Dec 3 2019, 1:16 PM

@Addshore says: it looks the script populating sites table on beta wikidata might have gotten run getting data from the wrong (non-beta) meta

WMDE-leszek added a comment.Dec 3 2019, 1:20 PM

First step: Investigation
Timebox: 2hrs
Quest: Find out what it is the right solution to this issue (Why there are non-existing wikis, and how to clean it up)

WMDE-leszek moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.Dec 3 2019, 1:20 PM
Stashbot added a comment.Dec 4 2019, 4:56 PM

Mentioned in SAL (#wikimedia-releng) [2019-12-04T16:56:38Z] <Amir1> MariaDB [wikidatawiki]> delete * from wb_changes_dispatch; (T237551)

Ladsgroup subscribed.Dec 4 2019, 5:01 PM

This was fun. The client, the list, everything for that was fine, the problem was that somehow wb_changes_dispatch table got production data (could have happened when someone mistakenly changes to $wgWBRepoSettings['localClientDatabases'] to be as the same as production. You can quickly fix it but when it's messes up the table, it can't get fixed on its own.) I dropped everything from that table and dispatching is back to normal now.

The possible solution is that SqlChangeDispatchCoordinator::getCandidateClients(), filter out things that are not in the its know client list. Fixing it should be easy.

Ladsgroup moved this task from Ready to pick up to Ready to estimate on the Wikidata-Campsite board.Dec 4 2019, 5:01 PM
Addshore closed this task as Resolved.Dec 5 2019, 10:12 AM
Addshore claimed this task.
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

Going to close this as it is resolved.

Restricted Application added a project: User-Addshore. · View Herald TranscriptDec 5 2019, 10:12 AM
Addshore moved this task from To Do (prioritised from top to bottom) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 5 2019, 10:12 AM
Ladsgroup claimed this task.Dec 5 2019, 10:14 AM

:D

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 5 2019, 10:14 AM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Dec 5 2019, 10:15 AM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.
Addshore added a comment.Dec 5 2019, 10:25 AM

It looks like although this issue is solved, this has exposed another issue, I will file a followup...

Addshore mentioned this in T239894: Dispatching broken on beta - Fatal error: Class 'Memcached' not found in ObjectCache.php on line 186.Dec 5 2019, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL