" route for MediaWiki where safe
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	jijiki
	Oct 5 2020, 12:11 PM

Description

As we had discussed previously, there are keys which we would like them to use the WarmUpRoute so to relieve some of the network traffic between our app servers and our memcached cluster.

SRE has created and deployed a mcrouter route called /*/mw-with-onhost-tier, which right now is configured exactly like the /*/mw route. The configuration which will route /*/mw-with-onhost-tier to use the WarmUpRoute can be enabled per MediaWiki server by setting profile::mediawiki::mcrouter_wancache::use_onhost_memcached to true.

We now need to have MediaWiki add the /*/mw-with-onhost-tier prefix to keys we want to be read from the onhost memcached, for instance the WANObjectCache keys. After this is deployed to MediaWiki, we can start gradually roll out to all clusters and test it in production.

Details

Subject	Repo	Branch	Lines +/-
mc: Remove mcrouter-with-onhost-tier from ParserCache	operations/mediawiki-config	master	+2 -6
objectcache: remove "onHostRoutingPrefix" feature from WANObjectCache	mediawiki/core	master	+5 -93
[Beta Cluster] mc-labs.php: Remove onHostRoutingPrefix for WAN cache	operations/mediawiki-config	master	+0 -3
[Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache	operations/mediawiki-config	master	+3 -0
[Beta Cluster] Fix undefined 'mcrouter-with-onhost-tier'	operations/mediawiki-config	master	+7 -4
[Beta Cluster] Enable onhost memc tier for ParserCache	operations/mediawiki-config	master	+3 -1
objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache	mediawiki/core	REL1_36	+360 -197
objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache	mediawiki/core	master	+360 -197
Make "mcrouter-with-onhost-tier" cache use "$wmfDatacenter"	operations/mediawiki-config	master	+1 -1
Switch parser cache to using "mcrouter-with-onhost-tier"	operations/mediawiki-config	master	+1 -1
Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches	operations/mediawiki-config	master	+4 -0

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T244852 Upgrade and improve our application object caching service (memcached)
Resolved	jijiki	T244340 Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached)
Declined	aaron	T264604 Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe
Resolved	jijiki	T270315 Upgrade memcached to version 1.6.x
Resolved	jijiki	T273115 Migrate onhost memcached to use a unix socket
Declined	aaron	T286531 Make WANObjectCache handle "lowTTL"/checkKeys/touchedCallback with "onHostRoutingPrefix" properly

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2020, 12:11 PM

jijiki updated the task description. (Show Details)Oct 5 2020, 12:12 PM

jijiki added a parent task: T244340: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached).

jijiki added a subscriber: Krinkle.

• Gilles assigned this task to aaron.Oct 5 2020, 6:36 PM

• Gilles moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.

aaron triaged this task as Medium priority.Oct 5 2020, 6:36 PM

Aklapper renamed this task from MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ to MediaWiki to route specific keys to /*/mw-with-onhost-tier/.Oct 5 2020, 7:36 PM

jijiki moved this task from Inbox 🐅 to Radar 📻 on the User-jijiki board.Oct 20 2020, 10:50 AM

Change 636094 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches

https://gerrit.wikimedia.org/r/636094

Change 636095 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Switch parser cache to using "mcrouter-with-onhost-tier"

https://gerrit.wikimedia.org/r/636095

@aaron is there a timeline as to when those patches will be merged?

In T264604#6601576, @jijiki wrote:

@aaron is there a timeline as to when those patches will be merged?

I was hoping it would get some CR. Is SRE OK with the patch? I assume a standard mwdebug => test => everything deploy will suffice. Does someone from SRE want to do the final sanity testing before the full scap sync? I could do some ParserCache testing in eval.php and by browsing around.

@aaron mwdebug1001 has the mcrouter configuration we want to roll out when we merge the mediawiki patches, so you are free to test it there. Meanwhile I will check with serviceops to get some CRs for your patches. Thank you!

jijiki moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Nov 6 2020, 1:04 PM

@aaron We will merge your patches on Monday and enable onhost memcached on an API canary host :)

Change 636094 merged by jenkins-bot:
[operations/mediawiki-config@master] Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches

https://gerrit.wikimedia.org/r/636094

Mentioned in SAL (#wikimedia-operations) [2020-11-10T12:23:52Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/mc.php: Config: [[gerrit:636094|Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches (T264604)]] (duration: 00m 57s)

Change 636095 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch parser cache to using "mcrouter-with-onhost-tier"

https://gerrit.wikimedia.org/r/636095

Mentioned in SAL (#wikimedia-operations) [2020-11-10T12:31:10Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:636095|Switch parser cache to using "mcrouter-with-onhost-tier" (T264604)]] (duration: 00m 57s)

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2020, 1:10 PM

@aaron @Krinkle We have successfully rolled this out on all app and api servers, and I am planning to continue with the jobrunners and parsoid.

Please let me know how would you like to continue this work and how I can help!

api:
memcached api

memcached slabs api

app servers:
memcached app

memcached app slabs

Thank you!

jijiki added a subscriber: Ladsgroup.Nov 24 2020, 8:21 PM

Can you point me to where (in Puppet?) the ttl enforcement and route/command filter for on-host reside? I'd like to link it from docs and maybe edit it to also link back to MW docs so that it doesn't get accidentally changed in ways that might (sublty) break compat.

@Krinkle would it help if I paste the generated config ?

The puppet part is here: mcrouter_wancache.pp#L114

jijiki added a subscriber: • Pchelolo.Nov 24 2020, 9:28 PM

Krinkle renamed this task from MediaWiki to route specific keys to /*/mw-with-onhost-tier/ to Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe.Nov 25 2020, 1:01 AM

Change 643384 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[operations/mediawiki-config@master] Make "mcrouter-with-onhost-tier" cache use "$wmfDatacenter"

https://gerrit.wikimedia.org/r/643384

gerritbot added a project: Patch-For-Review.Nov 25 2020, 1:03 AM

Change 643384 merged by jenkins-bot:
[operations/mediawiki-config@master] Make "mcrouter-with-onhost-tier" cache use "$wmfDatacenter"

https://gerrit.wikimedia.org/r/643384

@Krinkle @aaron do you think we are ready to move this forward?

In T264604#6681125, @jijiki wrote:

@Krinkle @aaron do you think we are ready to move this forward?

Beyond the parser cache? Anything else is probably blocked on https://phabricator.wikimedia.org/T252564 (for simplicity).

jijiki mentioned this in T273115: Migrate onhost memcached to use a unix socket.Jan 27 2021, 9:12 PM

@aaron now that T252564 has been unblocked, after I finish with T273115, I think we should proceed with moving this task forward

Next steps:

Update mediawiki/WANObjectCache to implement a new config option that lets you route "value" keys differently with a different mcrouter prefix.
Update mediawiki/WANObjectCache to store tombstone stored as sister-key (otherwise they will get trapped in the on-host tier as values).

I'll start with the first one of these.

jijiki added subtasks: T270315: Upgrade memcached to version 1.6.x, T273115: Migrate onhost memcached to use a unix socket.Mar 1 2021, 10:00 PM

jijiki awarded a token.

Krinkle moved this task from Doing (old) to Doing: Goals on the Performance-Team board.Mar 2 2021, 8:55 PM

Hm.. so the above poses a bit of a paradox. Implementing the onHostRoutingPrefix option is dependent on tombstones having their own sister-key. Implementing tombstones as their own sister-key, assuming we want that to be a feature flag for efficiency reasons, seems natural to do based on the onHostRoutingPrefix option.

So... I guess we could do it as one change then, I'll go with that for now.

Change 672514 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] objectcache: Implement 'onhostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/672514

jijiki mentioned this in T278220: Define the size of a pod for mediawiki in terms of resource usage.Mar 23 2021, 1:54 PM

jijiki closed subtask T270315: Upgrade memcached to version 1.6.x as Resolved.Mar 24 2021, 5:33 PM

Change 672514 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/672514

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.1; 2021-04-13).Apr 9 2021, 6:00 PM

jijiki closed subtask T273115: Migrate onhost memcached to use a unix socket as Resolved.Apr 14 2021, 11:10 AM

jijiki reopened subtask T270315: Upgrade memcached to version 1.6.x as Open.Apr 23 2021, 3:21 PM

Change 682698 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@REL1_36] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/682698

Change 682698 merged by jenkins-bot:

[mediawiki/core@REL1_36] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/682698

ReleaseTaggerBot added a project: MW-1.36-notes.Apr 26 2021, 7:00 PM

Change 691278 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] Enable onhost memc tier for ParserCache

https://gerrit.wikimedia.org/r/691278

Change 691278 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] Enable onhost memc tier for ParserCache

https://gerrit.wikimedia.org/r/691278

Change 691285 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] Fix undefined 'mcrouter-with-onhost-tier'

https://gerrit.wikimedia.org/r/691285

Change 691285 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] Fix undefined 'mcrouter-with-onhost-tier'

https://gerrit.wikimedia.org/r/691285

Change 692729 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/692729

• dpifke subscribed.Jun 7 2021, 6:12 PM

Change 692729 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/692729

Based on very rudimentary testing it seems that there is quite a direct correlation between the number of (concurrent) requests I make within any given 10 second window to the Main Page in beta, and the number of memc gets/s queries that make it through to the main memc cluster in beta. I would expect the on-host tier to absorb repeat queries within a given 10 second window. Maybe there are a lot of non-wan queries for the Main Page url, or maybe this is normal for other reasons, but it's worth checking a bit more before we proceed.

Logstash is currently broken in beta (T233134), but maybe we can workaround that by running tcpdump on appserver and memc servers and get an understanding of the traffic that way.

How many appserver instances are active?

Client

12 local tabs with:

$ watch -n0 "curl 'https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page' -H 'X-Wikimedia-Debug: 1' -I"

All responses have

server: deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
age: 0
x-cache: deployment-cache-text06 miss, deployment-cache-text06 pass
…
x-request-id: YNTMAD7WQOJ6ob0XBM2BcQAAAAM (constantly changing)
backend-timing: D=524558 t=1624558592089988 (constantly changing)
…

On-host tier

There is only one appserver in beta, deployment-mediawiki11. It has mcrouter on port 11213, and local memcached on 11210. It does not have anything running on the standard memc port.

krinkle@deployment-mediawiki11:~$ sudo memkeys -i lo -p 11210

memcache key	calls	objsize	req/sec
`WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv`	3581	53	12.88
`WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2f410ǀ#ǀv`	2973	1178	10.40
`WANCache:enwiki:messages:en:hash:v1ǀ#ǀv`	2954	124	10.19
`WANCache:enwiki:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6ǀ#ǀv`	2907	301	10.09
`WANCache:enwiki:sidebar:enǀ#ǀv`	2896	1046	10.09
`WANCache:global:resourceloader-titleinfo:enwiki:da39a3ee5e6b4b0d3255bfef95601890afd80709ǀ#ǀv`	2885	59	10.05
`WANCache:global:resourceloader-titleinfo:enwiki:115fe0f76f29a01e4c14c38fa78b43e2429d8d93ǀ#ǀv`	2875	577	10.02
`WANCache:global:NameTableSqlStore:wbt_type:wikidatawikiǀ#ǀv`	2873	117	9.94
`WANCache:enwiki:page-restrictions:v1:1:484119ǀ#ǀv`	2870	458	10.71
`WANCache:global:centralauth-user:32b41a9a3b733d7a3d196862daedf4a6ǀ#ǀv`	2868	5157	10.51
`WANCache:enwiki:page-content-model:484119ǀ#ǀv`	2868	71	10.70
`WANCache:global:centralauth-user:2792ced9d80ae274c2365b33e9f7f3b0ǀ#ǀv`	2868	3437	10.70
`enwiki:pcache:idoptions:1`	2765	664	11.15

Backend 1/3

krinkle@deployment-memc08:~$ sudo memkeys -i eth0

memcache key	calls	objsize	req/sec
`WANCache:wikidatawiki:messages:enǀ#ǀt`	207	25	0.33
`WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2f410ǀ#ǀv`	68	1178	0.11
`WANCache:global:resourceloader-titleinfo:enwiki:da39a3ee5e6b4b0d3255bfef95601890afd80709ǀ#ǀv`	68	59	0.11
`WANCache:global:centralauth-user:af8bbeb5e7e515bc8511e1d0030d3885ǀ#ǀv`	66	3459	0.11
`WANCache:enwiki:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6ǀ#ǀv`	65	301	0.10
`WANCache:global:centralauth-user:5bf1945342e67799cb50704a7fa19ac6ǀ#ǀv`	64	4109	0.11
`WANCache:global:centralauth-user:e9841ba61e91a7a4e31ecb105d67588aǀ#ǀv`	63	386	0.11

Backend 2/3

krinkle@deployment-memc09:~$ sudo memkeys -i eth0

memcache key	calls	objsize	req/sec
`WANCache:enwiki:gadgets-definition:9:2ǀ#ǀt`	7581	25	10.69
`WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv`	5211	54	7.81
`WANCache:wikidatawiki:gadgets-definition:9:2ǀ#ǀt`	385	25	0.54
`WANCache:loginwiki:messages:enǀ#ǀt`	332	25	0.47
`WANCache:global:NameTableSqlStore:content_models:wikidatawikiǀ#ǀv`	177	307	0.25
`WANCache:metawiki:messages:enǀ#ǀt`	119	25	0.17
`WANCache:global:centralauth-user:2792ced9d80ae274c2365b33e9f7f3b0ǀ#ǀv`	76	3437	0.11
`WANCache:enwiki:messages:en:hash:v1ǀ#ǀv`	76	124	0.11
`enwiki:pcache:idoptions:1`	72	664	0.10

Analysis

The keys going through on-host tier are as expected (only WANCache-value keys and parsercache keys).
The volume of requests through on-tier is as expected (for most keys it is the most or all of their reqs).
The keys going through to backends is as expected (same keys, plus non-eligiable keys such as WANCache-timestamp keys).
The volume of requests going through to backends is not entirely as expected.

For most keys that are eligible for onhost-tier we see, as expected, under 1 per second on the backend. But a notable exception is WANCache:global:rdbms-server-readonly|…|#|v and we're not sure why that it. It has 12 per second on the onhost tier, which seems about right (nearly all). Yet, we see an additional 5 per second on the backend, and I can't explain why that happens.

So the problem appears to be bad interactions between WANCache's "pre-emptive regeneration" feature (as prompted by lowTTL, which is enabled by default) and the fact that (naturally) values are kept constant for 10 seconds on the local memcached even if they are nearing the expiry and a new value has been generated already by a pre-emptive refresh.

The problem is that pre-emptive refreshes become increasingly more likely the closer we get to the expiry date. So if we store a value for 1 hour, then during the first 59 minutes everything is going fine in more-or-less 10 second windows through the on-host tier. In the last minute, however, we're going to eventually be 5 seconds away from the underlying value expiring (becoming boolean false).

During that 5 second window before the expiry on the main cluster, the app servers are going to increasingly more likely act as if the key has already expired and oppertunistically regenerate it. Since the local TTL of 10s is longer than 5s, none of them are succeeding in changing the value the next request sees, and this probability is keeps ramping up with basically all app servers acting as if memcached is down.

The key we see in our above analysis (rdbms-server-readonly) show-cases this problem beautifully since it has an expiry of only 5 seconds in total. Hence it continuously demonstrates this issue. This is kind of a red-herring since we could trivially decide to mitigate this in some other way by opting out of onhost routes for shortly lived keys, or do something else with them. But, the underlying issue that it helped us discover does appear to be real and affect all keys.

aaron mentioned this in T286531: Make WANObjectCache handle "lowTTL"/checkKeys/touchedCallback with "onHostRoutingPrefix" properly.Jul 13 2021, 3:09 AM

Krinkle added a subtask: T286531: Make WANObjectCache handle "lowTTL"/checkKeys/touchedCallback with "onHostRoutingPrefix" properly.Jul 19 2021, 4:09 PM

aaron closed subtask T286531: Make WANObjectCache handle "lowTTL"/checkKeys/touchedCallback with "onHostRoutingPrefix" properly as Declined.Oct 18 2021, 6:44 PM

Change 731817 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Remove onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/731817

Change 731817 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Remove onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/731817

Signing over to Aaron for removing some bits and pieces from WANObjectCache, which I'll review.

Change 731793 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] objectcache: remove \"onHostRoutingPrefix\" feature from WANObjectCache

https://gerrit.wikimedia.org/r/731793

Change 731793 merged by jenkins-bot:

[mediawiki/core@master] objectcache: remove \"onHostRoutingPrefix\" feature from WANObjectCache

https://gerrit.wikimedia.org/r/731793

aaron closed this task as Declined.Nov 2 2021, 3:40 AM

jijiki closed subtask T270315: Upgrade memcached to version 1.6.x as Resolved.Jan 10 2023, 8:51 PM

Change 937197 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] mc: Remove mcrouter-with-onhost-tier from ParserCache

https://gerrit.wikimedia.org/r/937197

Change 937197 merged by jenkins-bot:

[operations/mediawiki-config@master] mc: Remove mcrouter-with-onhost-tier from ParserCache

https://gerrit.wikimedia.org/r/937197

Mentioned in SAL (#wikimedia-operations) [2023-08-07T18:12:43Z] <krinkle@deploy1002> Started scap: Backport for [[gerrit:937197|mc: Remove mcrouter-with-onhost-tier from ParserCache (T264604)]]

Mentioned in SAL (#wikimedia-operations) [2023-08-07T18:14:06Z] <krinkle@deploy1002> krinkle: Backport for [[gerrit:937197|mc: Remove mcrouter-with-onhost-tier from ParserCache (T264604)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-08-07T18:21:50Z] <krinkle@deploy1002> Finished scap: Backport for [[gerrit:937197|mc: Remove mcrouter-with-onhost-tier from ParserCache (T264604)]] (duration: 09m 07s)

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 6:32 PM

jijiki mentioned this in T345740: Sunset onhost memcached on mediawiki servers and puppet.Sep 6 2023, 1:45 PM

Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safeClosed, DeclinedPublicActions