Page MenuHomePhabricator
Create Task
Maniphest T366857

InvalidArgumentException from line 80 of ServerInfo.php: No server with index '0' (in a maintenance script)
Open, LowPublicPRODUCTION ERROR
Actions

Description

While running maintenance scripts for T315510, we saw this error twice:

T315510#9575955

[0b0535de3852dcb0b1df256c] [no req]   InvalidArgumentException: No server with index '0'
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/ServerInfo.php(80)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(integer)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(908): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(integer, Wikimedia\Rdbms\DatabaseDomain, array)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(775): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(767): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#4 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#5 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#6 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(741): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#8 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(552): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#9 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(490): MediaWiki\Extension\DiscussionTools\ThreadItemStore::MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(564): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertIdButTryHarder(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)
#11 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(210): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)
#12 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#13 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#14 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#15 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#16 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#17 {main}

P64214

InvalidArgumentException from line 80 of /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/ServerInfo.php: No server with index '0'
#0 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1001): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(0)
#1 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(924): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(0, Object(Wikimedia\Rdbms\DatabaseDomain), Array)
#2 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(801): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(0, Object(Wikimedia\Rdbms\DatabaseDomain), 0)
#3 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793): Wikimedia\Rdbms\LoadBalancer->getServerConnection(0, 'enwiki', 0)
#4 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'enwiki', 0)
#5 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#6 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(420): Wikimedia\Rdbms\DBConnRef->__call('buildComparison', Array)
#7 /srv/mediawiki/php-1.43.0-wmf.6/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(157): Wikimedia\Rdbms\DBConnRef->buildComparison('>', Array)
#8 /srv/mediawiki/php-1.43.0-wmf.6/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Object(Wikimedia\Rdbms\SelectQueryBuilder), Array)
#9 /srv/mediawiki/php-1.43.0-wmf.6/maintenance/includes/MaintenanceRunner.php(696): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#10 /srv/mediawiki/php-1.43.0-wmf.6/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#11 /srv/mediawiki/multiversion/MWScript.php(158): require_once('/srv/mediawiki/...')
#12 {main}

Related Objects

Event Timeline

• matmarex created this task.Jun 6 2024, 10:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 6 2024, 10:55 PM
• matmarex mentioned this in T315510: Start maintenance script to backfill talk page comment database.Jun 6 2024, 10:56 PM
• matmarex added a subscriber: Ladsgroup.Jun 6 2024, 11:01 PM

Walking through that stack trace:

  • Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'enwiki', 0)
    • -1 is DB_REPLICA
  • ...
  • Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(0)
    • 0 is WRITER_INDEX.

I don't know this code but this is suspicious. Why is it looking at the writer when a replica was requested? And why does it not find it?

Krinkle moved this task from Untriaged to Rdbms library on the MediaWiki-libs-Rdbms board.Aug 30 2024, 1:06 AM
Krinkle added projects: Wikimedia-production-error, MediaWiki-API-Platform-Team.
Krinkle moved this task from Untriaged to Jun 2024 on the Wikimedia-production-error board.Sep 12 2024, 12:34 AM
BPirkle triaged this task as Medium priority.Sep 12 2024, 3:29 PM
BPirkle moved this task from Incoming (Needs Triage) to Backlog on the MediaWiki-API-Platform-Team board.
HCoplin-WMF lowered the priority of this task from Medium to Low.Oct 1 2024, 4:15 PM
aaron subscribed.Oct 1 2024, 4:20 PM

I haven't seen this error in logstash lately. I know that an error like this used to occur and was fixed in 65aa3c0a3e3. Presumably, someone changed the load weights in etcd while the script was running. This error shouldn't happen, though the script can be restarted if it does still occur.

@Ladsgroup, does this seem like something you want to look at?

HCoplin-WMF removed a project: MediaWiki-API-Platform-Team.Oct 1 2024, 4:29 PM
Ladsgroup added a comment.Oct 2 2024, 11:59 AM

I don't think this is fixed yet and it's not showing because we don't do many depool/repools now (dc switchover time). But I do think this is low priority.

Blake mentioned this in T416576: Make mw-cron jobs alert thresholds easily configurable.Feb 10 2026, 10:17 AM
dancy subscribed.Jun 2 2026, 3:00 PM

Here's a fresh report from a batch of these errors that I saw today:

Error
  • mwversion: 1.47.0-wmf.4
  • timestamp: 2026-06-02T14:57:08.780Z
  • phpversion: 8.3.31
  • reqId: 8ebce6dcdc4725b1f8a4ea87
  • Find reqId in Logstash
normalized_message
[{reqId}] {exception_url}   InvalidArgumentException: No server with index '0'
FrameLocationCall
from/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/ServerInfo.php(80)
#0/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(990)Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(int)
#1/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(909)Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(int, Wikimedia\Rdbms\DatabaseDomain, array)
#2/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(798)Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(int, Wikimedia\Rdbms\DatabaseDomain, int)
#3/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(1781)Wikimedia\Rdbms\LoadBalancer->getServerConnection(int, string, int)
#4/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/ObjectCache/WANObjectCache.php(1830)Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(bool, int, array, null, array)
#5/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/ObjectCache/WANObjectCache.php(1640)Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#6/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(1772)Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure, array)
#7/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/LoadBalancer/LoadBalancer.php(1758)Wikimedia\Rdbms\LoadBalancer->isPrimaryRunningReadOnly()
#8/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/Rdbms/ReadOnlyMode.php(55)Wikimedia\Rdbms\LoadBalancer->getReadOnlyReason()
#9/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/CentralAuthDatabaseManager.php(89)Wikimedia\Rdbms\ReadOnlyMode->getReason(string)
#10/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/CentralAuthDatabaseManager.php(72)MediaWiki\Extension\CentralAuth\CentralAuthDatabaseManager->getCentralReadOnlyReason()
#11/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(328)MediaWiki\Extension\CentralAuth\CentralAuthDatabaseManager->isReadOnly()
#12/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(547)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->shouldUsePrimaryDB()
#13/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(618)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->loadFromDatabase()
#14/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/ObjectCache/WANObjectCache.php(1830)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->MediaWiki\Extension\CentralAuth\User\{closure}(bool, int, array, null, array)
#15/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/libs/ObjectCache/WANObjectCache.php(1640)Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#16/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(611)Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure, array)
#17/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(473)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->loadFromCache()
#18/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(686)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->loadState()
#19/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthUser.php(804)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getId()
#20/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthIdLookup.php(331)MediaWiki\Extension\CentralAuth\User\CentralAuthUser->exists()
#21/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/CentralAuth/includes/User/CentralAuthIdLookup.php(303)MediaWiki\Extension\CentralAuth\User\CentralAuthIdLookup->isAttachedOn(MediaWiki\User\User, string, int)
#22/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/Echo/includes/NotifUser.php(651)MediaWiki\Extension\CentralAuth\User\CentralAuthIdLookup->centralIdFromLocalUser(MediaWiki\User\User, int)
#23/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/Echo/includes/NotifUser.php(430)MediaWiki\Extension\Notifications\NotifUser->getGlobalMemcKey(string)
#24/srv/mediawiki-staging/php-1.47.0-wmf.4/extensions/Echo/includes/Controller/ModerationController.php(44)MediaWiki\Extension\Notifications\NotifUser->resetNotificationCount()
#25/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/MWCallableUpdate.php(52)MediaWiki\Extension\Notifications\Controller\ModerationController::MediaWiki\Extension\Notifications\Controller\{closure}(string)
#26/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(448)MediaWiki\Deferred\MWCallableUpdate->doUpdate()
#27/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(187)MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\MWCallableUpdate)
#28/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(288)MediaWiki\Deferred\DeferredUpdates::run(MediaWiki\Deferred\MWCallableUpdate)
#29/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdatesScope.php(229)MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#30/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdatesScope.php(158)MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#31/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(285)MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#32/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdatesScope.php(229)MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#33/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdatesScope.php(158)MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#34/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(271)MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#35/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(363)MediaWiki\Deferred\DeferredUpdates::doUpdates(int)
#36/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Deferred/DeferredUpdates.php(142)MediaWiki\Deferred\DeferredUpdates::tryOpportunisticExecute()
#37/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Page/DeletePage.php(811)MediaWiki\Deferred\DeferredUpdates::addUpdate(MediaWiki\Deferred\SiteStatsUpdate)
#38/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Page/DeletePage.php(620)MediaWiki\Page\DeletePage->doDeleteUpdates(MediaWiki\Page\WikiPage, MediaWiki\Revision\RevisionStoreRecord)
#39/srv/mediawiki-staging/php-1.47.0-wmf.4/includes/Page/DeletePage.php(402)MediaWiki\Page\DeletePage->deleteInternal(MediaWiki\Page\WikiPage, string, string)
#40/srv/mediawiki-staging/php-1.47.0-wmf.4/maintenance/deleteBatch.php(118)MediaWiki\Page\DeletePage->deleteUnsafe(string)
#41/srv/mediawiki-staging/php-1.47.0-wmf.4/maintenance/includes/MaintenanceRunner.php(692)DeleteBatch->execute()
#42/srv/mediawiki-staging/php-1.47.0-wmf.4/maintenance/run.php(53)MediaWiki\Maintenance\MaintenanceRunner->run()
#43/srv/mediawiki-staging/multiversion/MWScript.php(219)require_once(string)
#44{main}
Impact/Notes

1929 hits in the last 6 hours.

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptJun 2 2026, 3:00 PM
• matmarex added a project: MediaWiki-Core-Platform-Team.Jun 8 2026, 1:51 PM
aaron added a comment.Jun 9 2026, 6:25 AM

The errors come from instances of the DeleteBatch maintenance script. It seems like a bunch of deferred updates built up during execution, then the primary was depooled and replaced, LoadBalancer::reconfigureServers() noticed and unset ->servers[0], and then the deferred updates tried to run. The try/catch logic in DeferredUpdatesScope went through each deferred update, with them all failing.

• matmarex edited projects, added: DBA; removed: MediaWiki-Core-Platform-Team.Jun 18 2026, 4:23 PM
Marostegui edited projects, added: Data-Persistence; removed: DBA.Jun 30 2026, 11:55 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits