I noticed that sometimes wtp hosts register mcrouter TKOs :
In the mcrouter logs it seems that the problematic shard is mc1027, more specifically slab 154 (you can see the peaks in the Weighted * graphs):
After a bit of tcpdump I found one key in the slab that might be the problem, namely: WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1
For example, traffic.pcap177 contains 2 seconds of mc1027's tx bandwidth traffic (from memcached to hosts) and this is the occurrence of the key:
tcpdump -r traffic.pcap177 -A | egrep 'WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1' | wc -l reading from file traffic.pcap177, link-type EN10MB (Ethernet) 87
All the occurrences are something like VALUE WANCache:v:nlwiki:preprocess-hash:6b35ce98ba6b117cf3eeb7807471739e:1 84 200702 (so the key's payload returned to the client from memcached).
Could it be a template expansion started a while ago?