We had an increase in our application servers traffic at ~19:40, which, possibly, overwhelmed memcached host mc1020. Since mcrouter was already flapping between the host and the gutter-pool, I firewalled the host until the event slowed down. We had alerts both for memcached and mediawiki errors.

Appservers GET latency

Increased TKOs

Network utilisation on mc1020

mc-gp1002 traffic

It appears that bw saturation started a bit earlier on this host, but not sure if related

Errors in Kibana:
https://logstash.wikimedia.org/goto/81369345cdbc9e9a34283ba26381df36

I also disabled puppet and downtimed memcacached on icinga, those changes need to be reverting in the EU morning. Firewalling probably helped, but I suspect that the traffic went down as well.