Page MenuHomePhabricator

beta wikidata trying to dispatch to wikis that don't exist
Closed, ResolvedPublic

Description

https://wikidata.beta.wmflabs.org/wiki/Special:DispatchStats

Dispatch statistics

SitePositionPendingLagTouched
Freshestaawiki167647100 minutes17:15, 6 November 2019
Mediancswikiversity166052315,94890 days, 8 hours and 4 minutes11:49, 7 August 2019
Stalestndswikiquote166052315,94890 days, 8 hours and 4 minutes11:50, 7 August 2019
Average--15,01285 days and 56 minutes-

But the median and stalest wikis don't exist in beta... So it shouldn't dispatch to them

As the "freshest" does exist, this might not be causing any actual problems. It just looks odd and presumably spamming the logs

reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki
17:15:48 Starting loop for unlimited passes or 540 seconds
17:15:51 ERROR: LogicException: Domain 'afwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('afwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
^C
reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki
17:16:06 Starting loop for unlimited passes or 540 seconds
17:16:06 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:21 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:33 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:16:44 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Failed to grab dispatch lock for afwiki
17:16:49 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:02 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:23 ERROR: LogicException: Domain 'azwikiquote' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('azwikiquote', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:37 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:17:48 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:03 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:14 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:26 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:37 ERROR: LogicException: Domain 'anwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('anwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:18:49 ERROR: LogicException: Domain 'abwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('abwiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
17:19:00 ERROR: LogicException: Domain 'acewiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196
Stack trace:
#0 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php(89): JobQueueGroup->lazyPush(Array)
#1 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/includes/ChangeDispatcher.php(243): Wikibase\Repo\Notifications\JobQueueChangeNotificationSender->sendNotification('acewiki', Array)
#2 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(302): Wikibase\Repo\ChangeDispatcher->dispatchTo(Array)
#3 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): Wikibase\DispatchChanges->execute()
#4 /srv/mediawiki-staging/php-master/extensions/Wikibase/repo/maintenance/dispatchChanges.php(399): require_once('/srv/mediawiki-...')
#5 /srv/mediawiki-staging/multiversion/MWScript.php(101): require_once('/srv/mediawiki-...')
#6 {main}; sleeping for 10 seconds
^C

Event Timeline

Reedy created this task.Nov 6 2019, 5:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2019, 5:25 PM
Addshore triaged this task as Medium priority.Sat, Nov 16, 9:15 PM
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

@Addshore says: it looks the script populating sites table on beta wikidata might have gotten run getting data from the wrong (non-beta) meta

First step: Investigation
Timebox: 2hrs
Quest: Find out what it is the right solution to this issue (Why there are non-existing wikis, and how to clean it up)

Mentioned in SAL (#wikimedia-releng) [2019-12-04T16:56:38Z] <Amir1> MariaDB [wikidatawiki]> delete * from wb_changes_dispatch; (T237551)

This was fun. The client, the list, everything for that was fine, the problem was that somehow wb_changes_dispatch table got production data (could have happened when someone mistakenly changes to $wgWBRepoSettings['localClientDatabases'] to be as the same as production. You can quickly fix it but when it's messes up the table, it can't get fixed on its own.) I dropped everything from that table and dispatching is back to normal now.

The possible solution is that SqlChangeDispatchCoordinator::getCandidateClients(), filter out things that are not in the its know client list. Fixing it should be easy.

Addshore closed this task as Resolved.Thu, Dec 5, 10:12 AM
Addshore claimed this task.

Going to close this as it is resolved.

Restricted Application added a project: User-Addshore. · View Herald TranscriptThu, Dec 5, 10:12 AM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptThu, Dec 5, 10:14 AM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.

It looks like although this issue is solved, this has exposed another issue, I will file a followup...