Page MenuHomePhabricator
Create Task
Maniphest T405087

Remove concept of groups in rdbms load balancer and replace it with shuffle sharding
Closed, ResolvedPublic
Actions

Description

The current concept of groups in rdbms load balancer is... bespoke. I have not seen it anywhere in the industry and it doesn't really isolate the load since DBAs have always been pooling api group replicas in the general group. One section (s3) has actually been operating without an api group for years now. The whole concept is causing all sorts of confusion and headache for us (DBAs) and it can hurt accuracy of the section-wide circuit breakers. Also, it is not clear what a group should refer to. api group is for API requests but vslow group is for slow queries in any type of requests.

My proposal is to use a more standard approach and simplify configuration:

  • Make rdbms library take a unique identifier, for users it can be something with session id, for logged out users it can be their IP.
  • Use that identifier and Amazon's shuffle sharding idea to pick the replica:
    • Use the identifier to pick two replicas out of the pool.
    • Based on weights, pick one of them
  • Keep vslow/dump group but use it as a flag and no need to explicitly define a replica in the config. Let rdbms with some magic consistently designate a replica as vslow.
  • Keep LoadMonitor, make sure the weights are properly reflected.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2025, 12:04 PM
gerritbot added a comment.Sep 23 2025, 11:12 PM

Change #1190794 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Introduce shuffle sharding to replace groups

https://gerrit.wikimedia.org/r/1190794

gerritbot added a project: Patch-For-Review.Sep 23 2025, 11:12 PM
aaron added a comment.EditedSep 24 2025, 11:52 PM

I like this idea. The current groups can also cause extra connections during requests (which then use their own REPEATABLE-READ view snapshot...). Assuming we phase out query groups and use this shuffle route approach, we'd avoid that problem, on top of simplifying things.

The original query groups idea was to:

  • Limit cache working set sizes by having different replica groups get different kinds of traffic. Of course, query groups like "api" apply to all manor of things reading different data, so it's probably not super helpful. There was also "recentchanges". Making it more fine grained is a burden and worsens the other problems (operations, extra connections, transaction isolation).
  • Use it to try out table partitioning schemes. We moved away from snowflake schemas, so that doesn't matter anymore.
  • I think query groups were also used waaay back to limit rdbms slowness/outage damage from bots to "just the API", which...has long since stopped making sense as a concept. Lots of our GUI even uses the API.

Since the API traffic is dominated by prolific bots, we'll get some working set size benefits if:

  • a bot frequently completes cycles of crawling through the same portion of data; relevant records are more likely to be in the buffer pool for those three mariadb servers. Related buffer pool churn is limited to those servers too.
  • a bot is doing an occasional crawl through a gigantic portion of data; at least any buffer pool churn only affects three mariadb servers.
  • if one bot does manage to slow down servers, despite query killers, it will be at least limited to three. Not something to really rely on, but it's still a plus. Cost-based rate limiting at the gateway will hopefully mitigate much of this.

Load groups for vslow make more sense, and they work on the entire request/script level, avoiding the extra connection problems. It still involves having to assign servers though. If it's done automatically, I assume that non-vslow requests/scripts would have to avoid those db servers. Automatic assignment would also have to not do anything dumb when there are large assigned load differences among db servers. I'm not sure of a clever way to do that.

Also, note that there is a HashRing class in /libs, though using the ArrayUtils method seems fine here, especially since we fix the readerIndex in LB.

Ladsgroup mentioned this in T403966: MariaDB Pre-DC switchover tasks.Sep 25 2025, 5:06 PM
CDanis subscribed.Sep 25 2025, 7:11 PM
Zabe subscribed.Sep 29 2025, 8:42 PM
gerritbot added a comment.Sep 29 2025, 9:08 PM

Change #1190794 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Introduce shuffle sharding to replace groups

https://gerrit.wikimedia.org/r/1190794

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2025, 9:32 PM
ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.21; 2025-09-30).Sep 29 2025, 10:00 PM
Ladsgroup moved this task from Triage to In progress on the DBA board.Oct 1 2025, 2:05 PM
gerritbot added a comment.Oct 2 2025, 1:08 PM

Change #1193108 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] db-production: Enable shuffle sharding

https://gerrit.wikimedia.org/r/1193108

gerritbot added a project: Patch-For-Review.Oct 2 2025, 1:08 PM
gerritbot added a comment.Oct 2 2025, 6:35 PM

Change #1193108 merged by jenkins-bot:

[operations/mediawiki-config@master] db-production: Enable shuffle sharding

https://gerrit.wikimedia.org/r/1193108

Stashbot added a comment.Oct 2 2025, 6:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T18:35:42Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1193108|db-production: Enable shuffle sharding (T405087)]]

Stashbot added a comment.Oct 2 2025, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T18:41:58Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1193108|db-production: Enable shuffle sharding (T405087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 2 2025, 6:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T18:58:14Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1193108|db-production: Enable shuffle sharding (T405087)]] (duration: 22m 32s)

Stashbot added a comment.Oct 2 2025, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T19:19:19Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s2 in eqiad (T405087)', diff saved to https://phabricator.wikimedia.org/P83576 and previous config saved to /var/cache/conftool/dbconfig/20251002-191918-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 7:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T19:27:28Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s2 in codfw (T405087)', diff saved to https://phabricator.wikimedia.org/P83577 and previous config saved to /var/cache/conftool/dbconfig/20251002-192726-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 7:29 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T19:29:29Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s5 and s6 in codfw (T405087)', diff saved to https://phabricator.wikimedia.org/P83578 and previous config saved to /var/cache/conftool/dbconfig/20251002-192928-ladsgroup.json

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2025, 7:30 PM
Stashbot added a comment.Oct 2 2025, 7:32 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T19:32:18Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s5 and s6 in eqiad (T405087)', diff saved to https://phabricator.wikimedia.org/P83579 and previous config saved to /var/cache/conftool/dbconfig/20251002-193217-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T19:54:27Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s7 in codfw (T405087)', diff saved to https://phabricator.wikimedia.org/P83580 and previous config saved to /var/cache/conftool/dbconfig/20251002-195426-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T20:01:44Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s7 in eqiad (T405087)', diff saved to https://phabricator.wikimedia.org/P83581 and previous config saved to /var/cache/conftool/dbconfig/20251002-200143-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 8:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T20:03:55Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s8 in eqiad (T405087)', diff saved to https://phabricator.wikimedia.org/P83582 and previous config saved to /var/cache/conftool/dbconfig/20251002-200354-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T20:06:21Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s8 in codfw (T405087)', diff saved to https://phabricator.wikimedia.org/P83583 and previous config saved to /var/cache/conftool/dbconfig/20251002-200621-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T20:15:33Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s4 and s1 in eqiad (T405087)', diff saved to https://phabricator.wikimedia.org/P83585 and previous config saved to /var/cache/conftool/dbconfig/20251002-201532-ladsgroup.json

Stashbot added a comment.Oct 2 2025, 8:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T20:25:37Z] <ladsgroup@cumin1003> dbctl commit (dc=all): 'Abolish api group from s4 and s1 in codfw (T405087)', diff saved to https://phabricator.wikimedia.org/P83587 and previous config saved to /var/cache/conftool/dbconfig/20251002-202536-ladsgroup.json

Ladsgroup added a comment.Oct 2 2025, 8:30 PM

There is no api group anymore.

gerritbot added a comment.Oct 6 2025, 11:47 AM

Change #1193823 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] api: Remove api group for picking the replica

https://gerrit.wikimedia.org/r/1193823

gerritbot added a project: Patch-For-Review.Oct 6 2025, 11:47 AM
gerritbot added a comment.Oct 6 2025, 1:48 PM

Change #1193823 merged by jenkins-bot:

[mediawiki/core@master] api: Remove api group for picking the replica

https://gerrit.wikimedia.org/r/1193823

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.22; 2025-10-07); removed MW-1.45-notes (1.45.0-wmf.21; 2025-09-30).Oct 6 2025, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 6 2025, 2:31 PM
gerritbot added a comment.Oct 7 2025, 11:32 AM

Change #1194177 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Fully remove $wgsetDefaultGroupName

https://gerrit.wikimedia.org/r/1194177

gerritbot added a project: Patch-For-Review.Oct 7 2025, 11:32 AM
gerritbot added a comment.Oct 7 2025, 11:37 AM

Change #1194179 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@master] repo: Remove dumpDBDefaultGroup config

https://gerrit.wikimedia.org/r/1194179

gerritbot added a comment.Oct 7 2025, 11:59 AM

Change #1194184 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Wikibase@master] Remove groups from DomainDb and co

https://gerrit.wikimedia.org/r/1194184

gerritbot added a comment.Oct 8 2025, 8:55 PM

Change #1194177 merged by jenkins-bot:

[mediawiki/core@master] Fully remove $wgDBDefaultGroup

https://gerrit.wikimedia.org/r/1194177

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.23; 2025-10-14); removed MW-1.45-notes (1.45.0-wmf.22; 2025-10-07).Oct 8 2025, 9:00 PM
Marostegui awarded a token.Oct 9 2025, 7:35 AM
Ladsgroup added a project: Wikidata.Oct 16 2025, 2:15 PM

Adding wikidata tag since we are now blocked on code reviews from WMDE.

gerritbot added a comment.Oct 20 2025, 1:44 PM

Change #1197269 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Deprecate any group beside vslow or dump

https://gerrit.wikimedia.org/r/1197269

gerritbot added a comment.Oct 20 2025, 2:34 PM

Change #1197277 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/CirrusSearch@master] Remove loadbalancer replica group

https://gerrit.wikimedia.org/r/1197277

gerritbot added a comment.Oct 20 2025, 2:39 PM

Change #1197280 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Echo@master] Stop using replica groups

https://gerrit.wikimedia.org/r/1197280

gerritbot added a comment.Oct 20 2025, 2:43 PM

Change #1194179 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] repo: Remove dumpDBDefaultGroup config

https://gerrit.wikimedia.org/r/1194179

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.24; 2025-10-21); removed MW-1.45-notes (1.45.0-wmf.23; 2025-10-14).Oct 20 2025, 3:00 PM
gerritbot added a comment.Oct 20 2025, 3:27 PM

Change #1194184 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Remove groups from DomainDb and co

https://gerrit.wikimedia.org/r/1194184

gerritbot added a comment.Oct 20 2025, 4:47 PM

Change #1197280 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Stop using replica groups

https://gerrit.wikimedia.org/r/1197280

gerritbot added a comment.Oct 20 2025, 8:42 PM

Change #1197277 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Remove loadbalancer replica group

https://gerrit.wikimedia.org/r/1197277

gerritbot added a comment.Oct 22 2025, 11:10 AM

Change #1197269 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Deprecate any group beside vslow or dump

https://gerrit.wikimedia.org/r/1197269

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2025, 11:31 AM
ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.25; 2025-10-28); removed MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 22 2025, 12:00 PM
gerritbot added a comment.Nov 14 2025, 4:40 PM

Change #1205190 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Dismantle concept of groups

https://gerrit.wikimedia.org/r/1205190

gerritbot added a project: Patch-For-Review.Nov 14 2025, 4:40 PM
gerritbot added a comment.Nov 18 2025, 3:31 PM

Change #1206897 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] rdbms: Use 'sectionLoads' name internally and improve LBFactory docs

https://gerrit.wikimedia.org/r/1206897

gerritbot added a comment.Nov 19 2025, 1:08 AM

Change #1205190 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Dismantle concept of groups

https://gerrit.wikimedia.org/r/1205190

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.4; 2025-11-25).Nov 19 2025, 2:00 AM
gerritbot added a comment.Nov 19 2025, 2:45 PM

Change #1207167 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.46.0-wmf.3] rdbms: Dismantle concept of groups

https://gerrit.wikimedia.org/r/1207167

gerritbot added a comment.Nov 19 2025, 3:20 PM

Change #1207167 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.3] rdbms: Dismantle concept of groups

https://gerrit.wikimedia.org/r/1207167

Stashbot added a comment.Nov 19 2025, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-19T15:22:09Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1207167|rdbms: Dismantle concept of groups (T405087)]]

Ladsgroup added a comment.Nov 19 2025, 3:26 PM
15:22:09 Updating ExtensionMessages-1.46.0-wmf.2.php
15:22:15 Updating LocalisationCache for 1.46.0-wmf.2 using 30 thread(s)
15:22:15 Running rebuildLocalisationCache.php
15:22:17 0 languages rebuilt out of 542
15:22:17 Use --force to rebuild the caches which are still fresh.
15:22:19 Generating JSON versions and md5 files (as www-data)
15:22:22 Updating ExtensionMessages-1.46.0-wmf.3.php
15:22:25 Warning: socket_sendto(): Unable to write to socket [101]: Network is unreachable in /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Debug/Logger/Monolog/LegacyHandler.php on line 220
[04b4b79e9a17d7b1365e43ff] [no req]   UnexpectedValueException: Section 'DEFAULT' has no hosts defined.
Backtrace:
from /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/libs/Rdbms/LBFactory/LBFactoryMulti.php(158)
#0 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/libs/Rdbms/LBFactory/LBFactoryMulti.php(208): Wikimedia\Rdbms\LBFactoryMulti->newMainLB(bool)
#1 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Storage/NameTableStoreFactory.php(97): Wikimedia\Rdbms\LBFactoryMulti->getMainLB(bool)
#2 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Storage/NameTableStoreFactory.php(118): MediaWiki\Storage\NameTableStoreFactory->get(string, bool)
#3 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/ServiceWiring.php(557): MediaWiki\Storage\NameTableStoreFactory->getChangeTagDef()
#4 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#5 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#6 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/MediaWikiServices.php(374): Wikimedia\Services\ServiceContainer->getService(string)
#7 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(414): MediaWiki\MediaWikiServices->getService(string)
#8 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(204): Wikimedia\Services\ServiceContainer->get(string)
#9 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(149): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#10 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(238): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#11 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(330): MediaWiki\HookContainer\HookContainer->makeExtensionHandlerCallback(string, array, array)
#12 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(502): MediaWiki\HookContainer\HookContainer->normalizeHandler(string, array, array)
#13 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(128): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#14 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookRunner.php(3926): MediaWiki\HookContainer\HookContainer->run(string, array)
#15 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Setup.php(468): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#16 /srv/mediawiki-staging/php-1.46.0-wmf.3/maintenance/doMaintenance.php(71): require_once(string)
#17 /srv/mediawiki-staging/php-1.46.0-wmf.3/maintenance/mergeMessageFileList.php(227): require_once(string)
#18 /srv/mediawiki-staging/multiversion/MWScript.php(221): require_once(string)
#19 {main}

15:22:25 Finished l10n-update (duration: 00m 17s)
15:22:25 sync-world failed: <CalledProcessError> Command '['sudo', '-u', 'mwbuilder', '-n', '--', '/usr/bin/scap', 'mwscript', '--no-local-config', '--directory', '/srv/mediawiki-staging', '--user', 'www-data', '--', 'mergeMessageFileList.php', '--wiki=aawiki', '--force-version', '1.46.0-wmf.3', '--list-file', '/srv/mediawiki-staging/wmf-config/extension-list', '--output', '/tmp/tmp.IxZM23pYxK']' returned non-zero exit status 255. (scap version: 4.227.0) (duration: 00m 20s)
15:22:25 backport failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-world', '--pause-after-testserver-sync', '--notify-user=ladsgroup', 'Backport for [[gerrit:1207167|rdbms: Dismantle concept of groups (T405087)]]']' returned non-zero exit status 1. (scap version: 4.227.0) (duration: 00m 54s)

I will debug this tomorrow.

Ladsgroup added a comment.Nov 19 2025, 3:47 PM

The error doesn't make any sense o.O

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.3; 2025-11-19); removed MW-1.46-notes (1.46.0-wmf.4; 2025-11-25).Nov 19 2025, 4:00 PM
Krinkle added a comment.EditedNov 19 2025, 5:07 PM

The groupLoadsBySection option is set in production but becomes unused after this patch. I suspect that might be what's defining DEFAULT in practice.

I suggest fully removing groupLoadsBySection from wmf-config and related (etcd, dbctl) to prove that it isn't needed. Be careful of course, because as-is that will probably take the site down since, unlike Scap, dbctl afaik won't run MediaWiki HTTP checks. So something may need to be added elsewhere first.

Ladsgroup added a comment.Nov 20 2025, 1:13 AM

But to my understanding currently in core the groupLoadsBySection array value gets fully ignored (after the merged patch) so it wouldn't be needed anymore (and I will clean it up soon) but it shouldn't cause an unconditional fatal either unless somehow the core patch impact other running versions but I can't imagine a way for that to happen. I'll debug this.

Ladsgroup added a comment.EditedNov 20 2025, 2:39 AM

Okay, I pulled the patch on mw-experimental, I double checked that the patch is applied to the code path I'm checking and double checked that I'm hitting the code path and opening the site, everything worked just fine. I even checked another wiki that was on another version to make sure. So either this was a network glitch that triggered other issues (e.g. by not allowing to access to etcd left the config dehydrated) and I panicked because the error looked related (very likely) or this happens on getMaintConnection or something like that (the error happens on a maint script) which is very unlikely. I'm gonna retry deploying the patch tomorrow. Let's see.

Ladsgroup added a comment.Nov 20 2025, 3:20 PM

Nope. Broke again :/

15:19:18 Updating ExtensionMessages-1.46.0-wmf.3.php
15:19:20 Warning: socket_sendto(): Unable to write to socket [101]: Network is unreachable in /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Debug/Logger/Monolog/LegacyHandler.php on line 220
[9f18a485ee7e921a99fbc7ee] [no req]   UnexpectedValueException: Section 'DEFAULT' has no hosts defined.
Backtrace:
from /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/libs/Rdbms/LBFactory/LBFactoryMulti.php(158)
#0 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/libs/Rdbms/LBFactory/LBFactoryMulti.php(208): Wikimedia\Rdbms\LBFactoryMulti->newMainLB(bool)
#1 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Storage/NameTableStoreFactory.php(97): Wikimedia\Rdbms\LBFactoryMulti->getMainLB(bool)
#2 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Storage/NameTableStoreFactory.php(118): MediaWiki\Storage\NameTableStoreFactory->get(string, bool)
#3 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/ServiceWiring.php(557): MediaWiki\Storage\NameTableStoreFactory->getChangeTagDef()
#4 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#5 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#6 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/MediaWikiServices.php(374): Wikimedia\Services\ServiceContainer->getService(string)
#7 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(414): MediaWiki\MediaWikiServices->getService(string)
#8 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(204): Wikimedia\Services\ServiceContainer->get(string)
#9 /srv/mediawiki-staging/php-1.46.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(149): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#10 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(238): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#11 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(330): MediaWiki\HookContainer\HookContainer->makeExtensionHandlerCallback(string, array, array)
#12 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(502): MediaWiki\HookContainer\HookContainer->normalizeHandler(string, array, array)
#13 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookContainer.php(128): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#14 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/HookContainer/HookRunner.php(3926): MediaWiki\HookContainer\HookContainer->run(string, array)
#15 /srv/mediawiki-staging/php-1.46.0-wmf.3/includes/Setup.php(468): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#16 /srv/mediawiki-staging/php-1.46.0-wmf.3/maintenance/doMaintenance.php(71): require_once(string)
#17 /srv/mediawiki-staging/php-1.46.0-wmf.3/maintenance/mergeMessageFileList.php(227): require_once(string)
#18 /srv/mediawiki-staging/multiversion/MWScript.php(221): require_once(string)
#19 {main}

I think it's something with maint scripts

Ladsgroup mentioned this in T410738: pretrain failing when calling mergeMessageFileList.php.Nov 21 2025, 2:31 PM
gerritbot added a comment.Nov 21 2025, 8:26 PM

Change #1208439 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Fix db config for offline maint scripts

https://gerrit.wikimedia.org/r/1208439

gerritbot added a comment.Nov 24 2025, 1:41 PM

Change #1208439 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix db config for offline maint scripts

https://gerrit.wikimedia.org/r/1208439

Stashbot added a comment.Nov 24 2025, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T13:41:54Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1208439|Fix db config for offline maint scripts (T410738 T405087)]]

Stashbot added a comment.Nov 24 2025, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T13:43:47Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1208439|Fix db config for offline maint scripts (T410738 T405087)]]

gerritbot added a comment.Nov 24 2025, 1:54 PM

Change #1210593 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Fix fix db config for offline maint scripts

https://gerrit.wikimedia.org/r/1210593

gerritbot added a comment.Nov 24 2025, 1:55 PM

Change #1210593 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix fix db config for offline maint scripts

https://gerrit.wikimedia.org/r/1210593

Stashbot added a comment.Nov 24 2025, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T13:55:54Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1208439|Fix db config for offline maint scripts (T410738 T405087)]], [[gerrit:1210593|Fix fix db config for offline maint scripts (T410738 T405087)]]

Stashbot added a comment.Nov 24 2025, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T14:00:36Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1208439|Fix db config for offline maint scripts (T410738 T405087)]], [[gerrit:1210593|Fix fix db config for offline maint scripts (T410738 T405087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Nov 24 2025, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T14:05:00Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1208439|Fix db config for offline maint scripts (T410738 T405087)]], [[gerrit:1210593|Fix fix db config for offline maint scripts (T410738 T405087)]] (duration: 09m 07s)

gerritbot added a comment.Nov 25 2025, 12:02 PM

Change #1211086 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@wmf/1.46.0-wmf.3] Revert^4 "rdbms: Dismantle concept of groups""

https://gerrit.wikimedia.org/r/1211086

gerritbot added a comment.Nov 25 2025, 12:18 PM

Change #1211086 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.3] Revert^4 "rdbms: Dismantle concept of groups""

https://gerrit.wikimedia.org/r/1211086

Stashbot added a comment.Nov 25 2025, 12:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-25T12:24:47Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1211086|Revert^4 "rdbms: Dismantle concept of groups"" (T405087)]]

Stashbot added a comment.Nov 25 2025, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-25T12:28:52Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1211086|Revert^4 "rdbms: Dismantle concept of groups"" (T405087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Nov 25 2025, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-25T12:34:55Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1211086|Revert^4 "rdbms: Dismantle concept of groups"" (T405087)]] (duration: 10m 08s)

Ladsgroup mentioned this in T411088: Clean up groups config.Nov 26 2025, 11:54 AM
Ladsgroup closed this task as Resolved.Dec 3 2025, 3:29 PM
Ladsgroup moved this task from In progress to Done on the DBA board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits