Page MenuHomePhabricator

Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe
Closed, DeclinedPublic

Description

As we had discussed previously, there are keys which we would like them to use the WarmUpRoute so to relieve some of the network traffic between our app servers and our memcached cluster.

SRE has created and deployed a mcrouter route called /*/mw-with-onhost-tier, which right now is configured exactly like the /*/mw route. The configuration which will route /*/mw-with-onhost-tier to use the WarmUpRoute can be enabled per MediaWiki server by setting profile::mediawiki::mcrouter_wancache::use_onhost_memcached to true.

We now need to have MediaWiki add the /*/mw-with-onhost-tier prefix to keys we want to be read from the onhost memcached, for instance the WANObjectCache keys. After this is deployed to MediaWiki, we can start gradually roll out to all clusters and test it in production.

Event Timeline

aaron triaged this task as Medium priority.Oct 5 2020, 6:36 PM
Aklapper renamed this task from MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ to MediaWiki to route specific keys to /*/mw-with-onhost-tier/.Oct 5 2020, 7:36 PM

Change 636094 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches

https://gerrit.wikimedia.org/r/636094

Change 636095 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Switch parser cache to using "mcrouter-with-onhost-tier"

https://gerrit.wikimedia.org/r/636095

@aaron is there a timeline as to when those patches will be merged?

@aaron is there a timeline as to when those patches will be merged?

I was hoping it would get some CR. Is SRE OK with the patch? I assume a standard mwdebug => test => everything deploy will suffice. Does someone from SRE want to do the final sanity testing before the full scap sync? I could do some ParserCache testing in eval.php and by browsing around.

@aaron mwdebug1001 has the mcrouter configuration we want to roll out when we merge the mediawiki patches, so you are free to test it there. Meanwhile I will check with serviceops to get some CRs for your patches. Thank you!

@aaron We will merge your patches on Monday and enable onhost memcached on an API canary host :)

Change 636094 merged by jenkins-bot:
[operations/mediawiki-config@master] Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches

https://gerrit.wikimedia.org/r/636094

Mentioned in SAL (#wikimedia-operations) [2020-11-10T12:23:52Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/mc.php: Config: [[gerrit:636094|Add "mcrouter-with-onhost-tier" entry to $wgObjectCaches (T264604)]] (duration: 00m 57s)

Change 636095 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch parser cache to using "mcrouter-with-onhost-tier"

https://gerrit.wikimedia.org/r/636095

Mentioned in SAL (#wikimedia-operations) [2020-11-10T12:31:10Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:636095|Switch parser cache to using "mcrouter-with-onhost-tier" (T264604)]] (duration: 00m 57s)

@aaron @Krinkle We have successfully rolled this out on all app and api servers, and I am planning to continue with the jobrunners and parsoid.

Please let me know how would you like to continue this work and how I can help!

api:
memcached api

memcached slabs api

app servers:
memcached app

memcached app slabs

Thank you!

Can you point me to where (in Puppet?) the ttl enforcement and route/command filter for on-host reside? I'd like to link it from docs and maybe edit it to also link back to MW docs so that it doesn't get accidentally changed in ways that might (sublty) break compat.

@Krinkle would it help if I paste the generated config ?

The puppet part is here: mcrouter_wancache.pp#L114

Krinkle renamed this task from MediaWiki to route specific keys to /*/mw-with-onhost-tier/ to Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe.Nov 25 2020, 1:01 AM

Change 643384 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[operations/mediawiki-config@master] Make "mcrouter-with-onhost-tier" cache use "$wmfDatacenter"

https://gerrit.wikimedia.org/r/643384

Change 643384 merged by jenkins-bot:
[operations/mediawiki-config@master] Make "mcrouter-with-onhost-tier" cache use "$wmfDatacenter"

https://gerrit.wikimedia.org/r/643384

@Krinkle @aaron do you think we are ready to move this forward?

@Krinkle @aaron do you think we are ready to move this forward?

Beyond the parser cache? Anything else is probably blocked on https://phabricator.wikimedia.org/T252564 (for simplicity).

@aaron now that T252564 has been unblocked, after I finish with T273115, I think we should proceed with moving this task forward

Next steps:

  1. Update mediawiki/WANObjectCache to implement a new config option that lets you route "value" keys differently with a different mcrouter prefix.
  2. Update mediawiki/WANObjectCache to store tombstone stored as sister-key (otherwise they will get trapped in the on-host tier as values).

I'll start with the first one of these.

Hm.. so the above poses a bit of a paradox. Implementing the onHostRoutingPrefix option is dependent on tombstones having their own sister-key. Implementing tombstones as their own sister-key, assuming we want that to be a feature flag for efficiency reasons, seems natural to do based on the onHostRoutingPrefix option.

So... I guess we could do it as one change then, I'll go with that for now.

Change 672514 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] objectcache: Implement 'onhostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/672514

Change 672514 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/672514

Change 682698 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@REL1_36] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/682698

Change 682698 merged by jenkins-bot:

[mediawiki/core@REL1_36] objectcache: Implement 'onHostRoutingPrefix' option in WANObjectCache

https://gerrit.wikimedia.org/r/682698

Change 691278 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] Enable onhost memc tier for ParserCache

https://gerrit.wikimedia.org/r/691278

Change 691278 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] Enable onhost memc tier for ParserCache

https://gerrit.wikimedia.org/r/691278

Change 691285 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] Fix undefined 'mcrouter-with-onhost-tier'

https://gerrit.wikimedia.org/r/691285

Change 691285 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] Fix undefined 'mcrouter-with-onhost-tier'

https://gerrit.wikimedia.org/r/691285

Change 692729 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/692729

Change 692729 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/692729

Based on very rudimentary testing it seems that there is quite a direct correlation between the number of (concurrent) requests I make within any given 10 second window to the Main Page in beta, and the number of memc gets/s queries that make it through to the main memc cluster in beta. I would expect the on-host tier to absorb repeat queries within a given 10 second window. Maybe there are a lot of non-wan queries for the Main Page url, or maybe this is normal for other reasons, but it's worth checking a bit more before we proceed.

Logstash is currently broken in beta (T233134), but maybe we can workaround that by running tcpdump on appserver and memc servers and get an understanding of the traffic that way.

How many appserver instances are active?

Client

12 local tabs with:

$ watch -n0 "curl 'https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page' -H 'X-Wikimedia-Debug: 1' -I"

All responses have

server: deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
age: 0
x-cache: deployment-cache-text06 miss, deployment-cache-text06 pass
…
x-request-id: YNTMAD7WQOJ6ob0XBM2BcQAAAAM (constantly changing)
backend-timing: D=524558 t=1624558592089988 (constantly changing)
…

On-host tier

There is only one appserver in beta, deployment-mediawiki11. It has mcrouter on port 11213, and local memcached on 11210. It does not have anything running on the standard memc port.

krinkle@deployment-mediawiki11:~$ sudo memkeys -i lo -p 11210
memcache keycallsobjsizereq/sec
WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv35815312.88
WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2f410ǀ#ǀv2973117810.40
WANCache:enwiki:messages:en:hash:v1ǀ#ǀv295412410.19
WANCache:enwiki:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6ǀ#ǀv290730110.09
WANCache:enwiki:sidebar:enǀ#ǀv2896104610.09
WANCache:global:resourceloader-titleinfo:enwiki:da39a3ee5e6b4b0d3255bfef95601890afd80709ǀ#ǀv28855910.05
WANCache:global:resourceloader-titleinfo:enwiki:115fe0f76f29a01e4c14c38fa78b43e2429d8d93ǀ#ǀv287557710.02
WANCache:global:NameTableSqlStore:wbt_type:wikidatawikiǀ#ǀv28731179.94
WANCache:enwiki:page-restrictions:v1:1:484119ǀ#ǀv287045810.71
WANCache:global:centralauth-user:32b41a9a3b733d7a3d196862daedf4a6ǀ#ǀv2868515710.51
WANCache:enwiki:page-content-model:484119ǀ#ǀv28687110.70
WANCache:global:centralauth-user:2792ced9d80ae274c2365b33e9f7f3b0ǀ#ǀv2868343710.70
enwiki:pcache:idoptions:1276566411.15

Backend 1/3

krinkle@deployment-memc08:~$ sudo memkeys -i eth0

memcache keycallsobjsizereq/sec
WANCache:wikidatawiki:messages:enǀ#ǀt207250.33
WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2f410ǀ#ǀv6811780.11
WANCache:global:resourceloader-titleinfo:enwiki:da39a3ee5e6b4b0d3255bfef95601890afd80709ǀ#ǀv68590.11
WANCache:global:centralauth-user:af8bbeb5e7e515bc8511e1d0030d3885ǀ#ǀv6634590.11
WANCache:enwiki:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6ǀ#ǀv653010.10
WANCache:global:centralauth-user:5bf1945342e67799cb50704a7fa19ac6ǀ#ǀv6441090.11
WANCache:global:centralauth-user:e9841ba61e91a7a4e31ecb105d67588aǀ#ǀv633860.11

Backend 2/3

krinkle@deployment-memc09:~$ sudo memkeys -i eth0

memcache keycallsobjsizereq/sec
WANCache:enwiki:gadgets-definition:9:2ǀ#ǀt75812510.69
WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv5211547.81
WANCache:wikidatawiki:gadgets-definition:9:2ǀ#ǀt385250.54
WANCache:loginwiki:messages:enǀ#ǀt332250.47
WANCache:global:NameTableSqlStore:content_models:wikidatawikiǀ#ǀv1773070.25
WANCache:metawiki:messages:enǀ#ǀt119250.17
WANCache:global:centralauth-user:2792ced9d80ae274c2365b33e9f7f3b0ǀ#ǀv7634370.11
WANCache:enwiki:messages:en:hash:v1ǀ#ǀv761240.11
enwiki:pcache:idoptions:1726640.10

Analysis

  • The keys going through on-host tier are as expected (only WANCache-value keys and parsercache keys).
  • The volume of requests through on-tier is as expected (for most keys it is the most or all of their reqs).
  • The keys going through to backends is as expected (same keys, plus non-eligiable keys such as WANCache-timestamp keys).
  • The volume of requests going through to backends is not entirely as expected.

For most keys that are eligible for onhost-tier we see, as expected, under 1 per second on the backend. But a notable exception is WANCache:global:rdbms-server-readonly|…|#|v and we're not sure why that it. It has 12 per second on the onhost tier, which seems about right (nearly all). Yet, we see an additional 5 per second on the backend, and I can't explain why that happens.

So the problem appears to be bad interactions between WANCache's "pre-emptive regeneration" feature (as prompted by lowTTL, which is enabled by default) and the fact that (naturally) values are kept constant for 10 seconds on the local memcached even if they are nearing the expiry and a new value has been generated already by a pre-emptive refresh.

The problem is that pre-emptive refreshes become increasingly more likely the closer we get to the expiry date. So if we store a value for 1 hour, then during the first 59 minutes everything is going fine in more-or-less 10 second windows through the on-host tier. In the last minute, however, we're going to eventually be 5 seconds away from the underlying value expiring (becoming boolean false).

During that 5 second window before the expiry on the main cluster, the app servers are going to increasingly more likely act as if the key has already expired and oppertunistically regenerate it. Since the local TTL of 10s is longer than 5s, none of them are succeeding in changing the value the next request sees, and this probability is keeps ramping up with basically all app servers acting as if memcached is down.

The key we see in our above analysis (rdbms-server-readonly) show-cases this problem beautifully since it has an expiry of only 5 seconds in total. Hence it continuously demonstrates this issue. This is kind of a red-herring since we could trivially decide to mitigate this in some other way by opting out of onhost routes for shortly lived keys, or do something else with them. But, the underlying issue that it helped us discover does appear to be real and affect all keys.

Change 731817 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Remove onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/731817

Change 731817 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta Cluster] mc-labs.php: Remove onHostRoutingPrefix for WAN cache

https://gerrit.wikimedia.org/r/731817

Signing over to Aaron for removing some bits and pieces from WANObjectCache, which I'll review.

Change 731793 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] objectcache: remove \"onHostRoutingPrefix\" feature from WANObjectCache

https://gerrit.wikimedia.org/r/731793

Change 731793 merged by jenkins-bot:

[mediawiki/core@master] objectcache: remove \"onHostRoutingPrefix\" feature from WANObjectCache

https://gerrit.wikimedia.org/r/731793