I noticed that sometimes wtp hosts register mcrouter TKOs :
https://grafana.wikimedia.org/d/000000549/mcrouter?orgId=1&fullscreen&panelId=9&from=now-24h&to=now
In the mcrouter logs it seems that the problematic shard is mc1027, more specifically slab 154 (you can see the peaks in the Weighted * graphs):
https://grafana.wikimedia.org/d/000000317/memcache-slabs?orgId=1&from=now-6h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=memcached&var-instance=mc1027&var-slab=All
After a bit of tcpdump I found one key in the slab that might be the problem, namely: `WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1`
For example, `traffic.pcap177` contains 2 seconds of tx bandwidth traffic (from memcached to hosts) and this is the occurrence of the key:
```
tcpdump -r traffic.pcap177 -A | egrep 'WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1' | wc -l
reading from file traffic.pcap177, link-type EN10MB (Ethernet)
87
```
All the occurrences are something like `VALUE WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1 84 200702` (so the key's payload returned to the client from memcached).
Could it be a template expansion started a while ago?