Page MenuHomePhabricator
Create Task
Maniphest T411807

WF memcached service is dc-local but used for dc-global content
Closed, ResolvedPublic
Actions

Description

[Split this specific issue out from the parent for clarity.]

mcrouter-wikifunctions is dc-local; values are written to local/wf, without fail-over. This is as the system was built in 2023, and the Abstract team (and SRE) didn't notice that the planned use was for implicitly dc-global content, feeding into Wikifunctions.org, and later the parser cache.

Options (please edit!):

  • Add a new cache service that's dc-global and shift use over to that and then decommission the old one
  • Retrofit somehow the existing service to work dc-global
  • Use the databases as rendered content caches instead of memcached
  • ???

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mw-mcrouter: add /{dc}/wf-wan routes for Wikifunctions client cacheoperations/deployment-chartsmaster+30 -0
mw-wikifunctions: Set $MCROUTER_SERVER in values-${ENV}.yamloperations/deployment-chartsmaster+6 -4
Wikifunctions: Switch cache from mcrouter-wikifunctions to special accessoperations/mediawiki-configmaster+15 -8
Revert "Wikifunctions: Switch cache from mcrouter-wikifunctions to special access"operations/mediawiki-configmaster+8 -15
cache.mcrouter: Add replica.remote_read optionoperations/deployment-chartsmaster+11 -5
WikiLambdaServices::buildZObjectStash: Pass in cache params to WANObjectCachemediawiki/extensions/WikiLambdamaster+4 -1
[DNM] memcached: Point to the replicated Wikifunctions cacheoperations/mediawiki-configmaster+3 -3
mc: Shift the Wikifunctions MC route from /local/wf/ to /<dc>/wf-wan/operations/mediawiki-configmaster+7 -3
wikifunctions and friends: Add /*/wf-wan memcache routesoperations/deployment-chartsmaster+43 -0
_mediawiki-common_: Add /*/wf-wan memcache routesoperations/deployment-chartsmaster+12 -0
mw-experimental: Add /*/wf-wan memcache routes and use in-pod mcrouteroperations/deployment-chartsmaster+62 -2
mw-debug: Add /*/wf-wan memcache routesoperations/deployment-chartsmaster+24 -0
[DNM] memcached: Drop the local-only Wikifunctions cache routeoperations/puppetproduction+2 -10
[WIP] mcrouter: Configure the Wikifunctions pool as replicatedoperations/puppetproduction+55 -1
mcrouter: Allow configuring secondary replicated cachesoperations/puppetproduction+27 -22
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.Dec 4 2025, 5:58 PM
Jdforrester-WMF triaged this task as Unbreak Now! priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2025, 5:58 PM
DSantamaria moved this task from Incoming to Ready on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Dec 5 2025, 11:15 AM
Kappakayala added subscribers: jijiki, Kappakayala.Dec 8 2025, 11:35 PM

@jijiki Could you please help look into it?

Kappakayala added a comment.Dec 10 2025, 7:13 PM

Alex @akosiaris briefly looked into this and this seems to be more than a bug at this time. He is planning to secure some time to consult some folks from Security and Mediawiki while he is in the offsite to discuss. He should be able to provide more details once those conversations are done.

akosiaris added a comment.Dec 11 2025, 10:31 AM

FYI, I 've continued posting updates in T405461

akosiaris mentioned this in T405461: Embedded function calls getting stuck showing "Function being called..." instead of result, due to (?) split-brain cache problem.Dec 13 2025, 11:10 AM
akosiaris lowered the priority of this task from Unbreak Now! to High.Dec 16 2025, 10:22 AM

Lowering to high while the analysis and recommendation is being discussed in T405461: Embedded function calls getting stuck showing "Function being called..." instead of result, due to (?) split-brain cache problem.

I also think we should merge this one in that parent task. Any objections?

Jdforrester-WMF added a comment.Jan 5 2026, 3:03 PM
In T411807#11463249, @akosiaris wrote:

I also think we should merge this one in that parent task. Any objections?

This was meant to be the cross-team discussion task, but never mind.

Jdforrester-WMF closed this task as a duplicate of T405461: Embedded function calls getting stuck showing "Function being called..." instead of result, due to (?) split-brain cache problem.Jan 5 2026, 3:03 PM
Jdforrester-WMF reopened this task as Open.Jan 21 2026, 12:26 AM

Re-opening as the doing-task for one approach that might (but might not) solve the parent.

Jdforrester-WMF changed the task status from Open to In Progress.Jan 21 2026, 12:27 AM
Jdforrester-WMF claimed this task.
Jdforrester-WMF edited projects, added Abstract Wikipedia team (26Q3 (Jan–Mar)); removed Abstract Wikipedia team (26Q2 (Oct–Dec)).
Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.
gerritbot added a comment.Jan 21 2026, 12:59 AM

Change #1229229 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/puppet@production] mcrouter: Allow configuring secondary replicated caches

https://gerrit.wikimedia.org/r/1229229

gerritbot added a project: Patch-For-Review.Jan 21 2026, 12:59 AM

Change #1229232 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/mediawiki-config@master] [DNM] memcached: Point to the replicated Wikifunctions cache

https://gerrit.wikimedia.org/r/1229232

gerritbot added a comment.Jan 21 2026, 12:59 AM

Change #1229230 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/puppet@production] [WIP] mcrouter: Configure the Wikifunctions pool as replicated

https://gerrit.wikimedia.org/r/1229230

gerritbot added a comment.Jan 21 2026, 12:59 AM

Change #1229231 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/puppet@production] [DNM] memcached: Drop the local-only Wikifunctions cache route

https://gerrit.wikimedia.org/r/1229231

MLechvien-WMF removed a project: serviceops-deprecated.Jan 25 2026, 10:26 PM
jijiki added a comment.Feb 5 2026, 11:07 AM

@Jdforrester-WMF to my understanding, you would like any mw-wf memcached keys we add in one DC, to be replicated to the other DC and vice versa?

jijiki added projects: ServiceOps-Datastores, ServiceOps new.Feb 5 2026, 11:11 AM
jijiki moved this task from Inbox to In Progress on the ServiceOps new board.
MLechvien-WMF moved this task from In Progress to Needs Info / Blocked on the ServiceOps new board.Feb 6 2026, 9:31 AM
Jdforrester-WMF added a comment.Feb 9 2026, 5:27 PM
In T411807#11586750, @jijiki wrote:

@Jdforrester-WMF to my understanding, you would like any mw-wf memcached keys we add in one DC, to be replicated to the other DC and vice versa?

Yes, with best-efforts fixes, to resolve the current split-brain problem that has been a blocker to KRs since it was identified in September. In discussions with Alex concluding in December and January this seemed to be the only acceptable way forward for SRE and Product, given that the cache is written to in both data-centres due to MW's active-active status and read from in both (and by many wikis, currently 150 and hopefully-soon > 300).

MLechvien-WMF moved this task from Needs Info / Blocked to In Progress on the ServiceOps new board.Feb 13 2026, 10:46 AM
MLechvien-WMF added subscribers: RLazarus, MLechvien-WMF.

@RLazarus is currently getting context/ramping up on this and will be able to support this work moving forward.

gerritbot added a comment.Feb 27 2026, 6:35 AM

Change #1245162 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] [WIP] wikifunctions and friends: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1245162

gerritbot added a comment.Mar 3 2026, 7:26 PM

Change #1229229 abandoned by Jforrester:

[operations/puppet@production] mcrouter: Allow configuring secondary replicated caches

Reason:

Doing this in deployment-charts instead now.

https://gerrit.wikimedia.org/r/1229229

gerritbot added a comment.Mar 3 2026, 7:26 PM

Change #1229230 abandoned by Jforrester:

[operations/puppet@production] [WIP] mcrouter: Configure the Wikifunctions pool as replicated

Reason:

Doing this in deployment-charts instead now.

https://gerrit.wikimedia.org/r/1229230

gerritbot added a comment.Mar 3 2026, 7:26 PM

Change #1229231 abandoned by Jforrester:

[operations/puppet@production] [DNM] memcached: Drop the local-only Wikifunctions cache route

Reason:

Doing this in deployment-charts instead now.

https://gerrit.wikimedia.org/r/1229231

gerritbot added a comment.Mar 3 2026, 9:01 PM

Change #1247677 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] _mediawiki-common_: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1247677

gerritbot added a comment.Mar 3 2026, 9:01 PM

Change #1247678 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] wikifunctions and friends: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1247678

gerritbot added a comment.Mar 3 2026, 9:45 PM

Change #1245162 merged by jenkins-bot:

[operations/deployment-charts@master] mw-debug: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1245162

Stashbot added a comment.Mar 3 2026, 9:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-03T21:54:18Z] <rzl@deploy2002> Started scap sync-world: https://gerrit.wikimedia.org/r/1245162 T411807

Stashbot added a comment.Mar 3 2026, 9:55 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-03T21:55:17Z] <rzl@deploy2002> rzl: https://gerrit.wikimedia.org/r/1245162 T411807 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 3 2026, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-03T21:59:07Z] <rzl@deploy2002> Finished scap sync-world: https://gerrit.wikimedia.org/r/1245162 T411807 (duration: 12m 15s)

gerritbot added a comment.Mar 3 2026, 10:09 PM

Change #1247687 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/mediawiki-config@master] mc: Shift the Wikifunctions MC route from /local/wf/ to /<dc>/wf-wan/

https://gerrit.wikimedia.org/r/1247687

gerritbot added a comment.Mar 3 2026, 10:29 PM

Change #1247694 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-experimental: Add /*/wf-wan memcache routes and use in-pod mcrouter

https://gerrit.wikimedia.org/r/1247694

gerritbot added a comment.Mar 3 2026, 11:05 PM

Change #1247694 merged by jenkins-bot:

[operations/deployment-charts@master] mw-experimental: Add /*/wf-wan memcache routes and use in-pod mcrouter

https://gerrit.wikimedia.org/r/1247694

Jdforrester-WMF added a subtask: T419666: WikiLambda: Replace direct usage of BagOStuff with WANObjectCache.Mar 11 2026, 1:27 PM
Jdforrester-WMF changed the status of subtask T419666: WikiLambda: Replace direct usage of BagOStuff with WANObjectCache from Open to In Progress.Mar 11 2026, 1:29 PM
gerritbot added a comment.Mon, Mar 16, 10:22 PM

Change #1247677 merged by jenkins-bot:

[operations/deployment-charts@master] _mediawiki-common_: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1247677

gerritbot added a comment.Mon, Mar 16, 10:23 PM

Change #1247678 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions and friends: Add /*/wf-wan memcache routes

https://gerrit.wikimedia.org/r/1247678

Stashbot added a comment.Mon, Mar 16, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-16T22:27:32Z] <jforrester@deploy2002> Started scap sync-world: T411807

Stashbot added a comment.Mon, Mar 16, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-16T22:28:10Z] <jforrester@deploy2002> jforrester: T411807 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mon, Mar 16, 10:37 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-16T22:37:00Z] <jforrester@deploy2002> Finished scap sync-world: T411807 (duration: 11m 10s)

gerritbot added a comment.Mon, Mar 16, 11:06 PM

Change #1229232 abandoned by Jforrester:

[operations/mediawiki-config@master] [DNM] memcached: Point to the replicated Wikifunctions cache

Reason:

For whatever reason I89738e16c3c92edd5d37ddf3e6042e518bdc2730 got pushed.

https://gerrit.wikimedia.org/r/1229232

gerritbot added a comment.Thu, Mar 19, 6:51 PM

Change #1255833 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] WikiLambdaServices::buildZObjectStash: Pass in cache params to WANObjectCache

https://gerrit.wikimedia.org/r/1255833

gengh closed subtask T419666: WikiLambda: Replace direct usage of BagOStuff with WANObjectCache as Resolved.Mon, Mar 23, 10:27 AM
gerritbot added a comment.Mon, Mar 23, 8:43 PM

Change #1259222 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] cache.mcrouter: Add replica.remote_read option

https://gerrit.wikimedia.org/r/1259222

gerritbot added a comment.Mon, Mar 30, 1:45 PM

Change #1264638 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/mediawiki-config@master] Revert "Wikifunctions: Switch cache from mcrouter-wikifunctions to special access"

https://gerrit.wikimedia.org/r/1264638

gerritbot added a comment.Mon, Mar 30, 1:46 PM

Change #1264638 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Wikifunctions: Switch cache from mcrouter-wikifunctions to special access"

https://gerrit.wikimedia.org/r/1264638

Maintenance_bot added a project: Abstract Wikipedia.Mon, Mar 30, 2:30 PM
gerritbot added a comment.Mon, Mar 30, 2:43 PM

Change #1255833 abandoned by Jforrester:

[mediawiki/extensions/WikiLambda@master] WikiLambdaServices::buildZObjectStash: Pass in cache params to WANObjectCache

https://gerrit.wikimedia.org/r/1255833

gerritbot added a comment.Wed, Apr 1, 2:59 PM

Change #1266290 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/mediawiki-config@master] Wikifunctions: Switch cache from mcrouter-wikifunctions to special access

https://gerrit.wikimedia.org/r/1266290

gerritbot added a comment.Wed, Apr 1, 3:00 PM

Change #1266290 merged by jenkins-bot:

[operations/mediawiki-config@master] Wikifunctions: Switch cache from mcrouter-wikifunctions to special access

https://gerrit.wikimedia.org/r/1266290

Stashbot added a comment.Wed, Apr 1, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:01:00Z] <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1266290|Wikifunctions: Switch cache from mcrouter-wikifunctions to special access (T411807)]]

Stashbot added a comment.Wed, Apr 1, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:02:59Z] <jforrester@deploy1003> jforrester: Backport for [[gerrit:1266290|Wikifunctions: Switch cache from mcrouter-wikifunctions to special access (T411807)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Wed, Apr 1, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:13:54Z] <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1266290|Wikifunctions: Switch cache from mcrouter-wikifunctions to special access (T411807)]] (duration: 12m 53s)

Jdforrester-WMF closed this task as Resolved.Wed, Apr 1, 3:17 PM
Jdforrester-WMF moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.

OK, I'm declaring this particular bit Resolved. Wikifunctions's use of memcached now (a) works consistently across DCs (content is triggered in one and read from the other, both ways around), and (b) uses our own dedicated caches again, not polluting MW's general cache.

There's much more work to do to re-assess our content secondary storage / availability strategy and use of memcached, DBs, and other technologies, but let's call this done.

gerritbot added a comment.Fri, Apr 3, 6:09 PM

Change #1267915 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-wikifunctions: Set $MCROUTER_SERVER in values-${ENV}.yaml

https://gerrit.wikimedia.org/r/1267915

gerritbot added a comment.Wed, Apr 8, 6:33 PM

Change #1269038 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] mw-mcrouter: add /{dc}/wf-wan routes for Wikifunctions client cache

https://gerrit.wikimedia.org/r/1269038

gerritbot added a comment.Wed, Apr 8, 6:52 PM

Change #1269038 merged by jenkins-bot:

[operations/deployment-charts@master] mw-mcrouter: add /{dc}/wf-wan routes for Wikifunctions client cache

https://gerrit.wikimedia.org/r/1269038

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits