Page MenuHomePhabricator

Occasional NIC Tx bandwidth saturation for mc1027
Closed, ResolvedPublic

Description

I noticed that sometimes wtp hosts register mcrouter TKOs :

https://grafana.wikimedia.org/d/000000549/mcrouter?orgId=1&fullscreen&panelId=9&from=now-24h&to=now

In the mcrouter logs it seems that the problematic shard is mc1027, more specifically slab 154 (you can see the peaks in the Weighted * graphs):

https://grafana.wikimedia.org/d/000000317/memcache-slabs?orgId=1&from=now-6h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=memcached&var-instance=mc1027&var-slab=All

After a bit of tcpdump I found one key in the slab that might be the problem, namely: WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1

For example, traffic.pcap177 contains 2 seconds of mc1027's tx bandwidth traffic (from memcached to hosts) and this is the occurrence of the key:

tcpdump -r traffic.pcap177 -A | egrep 'WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1' | wc -l
reading from file traffic.pcap177, link-type EN10MB (Ethernet)
      87

All the occurrences are something like VALUE WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1 84 200702 (so the key's payload returned to the client from memcached).

Could it be a template expansion started a while ago?

Event Timeline

elukey triaged this task as High priority.Mar 31 2020, 9:37 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2020, 9:37 AM
elukey updated the task description. (Show Details)Mar 31 2020, 9:37 AM
elukey added a subscriber: CDanis.
elukey added a comment.EditedApr 3 2020, 6:48 AM

@Krinkle @aaron Can you tell me some hints about how to figure out what is the key's purpose? It feels like a popular template change in nlwiki but I'd like to triple check. We are still seeing saturation problems.. :(

Edit: correcting myself, the current TKO events are related to other shards, this problem seems gone (even checking on slab metrics traffic it seems "fixed").

I am going to close this task as there are no good action items, and open a task to add more precise bandwidth usage monitoring.

elukey closed this task as Resolved.Apr 3 2020, 7:04 AM
elukey claimed this task.
aaron added a comment.Apr 3 2020, 4:55 PM

It stores the serialized naive "top frame" (e.g. headings, paragraphs, template invocation parameters) of the wikitext of pages, as well as the "sub-frames" from template invocations, upon recursive expansion. This all happens on page parse. Note that these keys do not need purges. If template X is invoked the same way on multiple pages, then parses of those pages will reuse a common sub-frame cache key for those template invocations and likewise for templates invoked from that template. So, I suppose a popular template invoked with a low enough cardinality of paramater/context bundles would trigger traffic spikes upon invalidation. The traffic would come from either (a) refreshLinks jobs or (b) page views to backlink pages that got purged via htmlCacheUpdate jobs.

This would be even worse for a ~1Mb key (instead of just ~200Kb). Is there any ETA on 10GB link upgrades?

elukey added a comment.Apr 7 2020, 9:27 AM

It stores the serialized naive "top frame" (e.g. headings, paragraphs, template invocation parameters) of the wikitext of pages, as well as the "sub-frames" from template invocations, upon recursive expansion. This all happens on page parse. Note that these keys do not need purges. If template X is invoked the same way on multiple pages, then parses of those pages will reuse a common sub-frame cache key for those template invocations and likewise for templates invoked from that template. So, I suppose a popular template invoked with a low enough cardinality of paramater/context bundles would trigger traffic spikes upon invalidation. The traffic would come from either (a) refreshLinks jobs or (b) page views to backlink pages that got purged via htmlCacheUpdate jobs.

This would be even worse for a ~1Mb key (instead of just ~200Kb). Is there any ETA on 10GB link upgrades?

Thanks a lot for the explanation, then my theory makes sense (IIUC). There is not ETA for 10G upgrades, but we shouldn't be far from having the gutter pool ready (with 10g links)..

aaron added a comment.Apr 7 2020, 6:25 PM

I wonder if it would be useful for the template name to appear in the key when possible. Right now it's just an opaque hash. I doubt that many invocations of different templates have the same top-level text, so I don't see it adding much fragmentation. Maybe some statsd logging could be done instead though.

aaron added a comment.Apr 7 2020, 10:29 PM

Hmm, it would help if cacheGetTree() /cacheSetTree() were replaced by getWithSetCallback() perhaps. Lots of optimizations are not used atm due to that fact.

elukey added a comment.Apr 8 2020, 6:52 AM

@aaron one thing that it would be useful is, in my opinion, having instrumentation in MediaWiki about key size volume/bytes. Even per "key family" would be enough, just to spot regressions in bandwidth usage from grafana rather than tracking them down via tcpdump. Do you think that it would be possible?

aaron added a comment.Apr 9 2020, 8:40 PM

@aaron one thing that it would be useful is, in my opinion, having instrumentation in MediaWiki about key size volume/bytes. Even per "key family" would be enough, just to spot regressions in bandwidth usage from grafana rather than tracking them down via tcpdump. Do you think that it would be possible?

It's a lot easier than it used to be. First, I'll try to make the preprocessor use getWithSetCallback().

Change 587902 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: add size metrics to WANObjectCache::getWithSetCallback()

https://gerrit.wikimedia.org/r/587902

Change 587902 merged by jenkins-bot:
[mediawiki/core@master] objectcache: add size metrics to WANObjectCache::getWithSetCallback()

https://gerrit.wikimedia.org/r/587902

Change 591237 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: store the serialized size estimates alongside values

https://gerrit.wikimedia.org/r/591237