After cleaning up all the noise related to high spikes of TKOs, I noticed that sometimes we still register little spikes (10~100 requests affected max) related to codfw proxies. For example:
Jul 4 00:28:58 mw1235 mcrouter: I0704 00:28:58.840715 13476 AsyncSSLSocket.cpp:119] TCP connect failed: AsyncSocketException: connect timed out after 1000ms, type = Timed out Jul 4 00:28:58 mw1235 mcrouter: I0704 00:28:58.840767 13476 ProxyDestination.cpp:453] 10.192.0.61:11214 marked hard TKO. Total hard TKOs: 1; soft TKOs: 0. Reply: mc_res_connect_error Jul 4 00:29:02 mw1235 mcrouter: I0704 00:29:02.877496 13476 ProxyDestination.cpp:453] 10.192.0.61:11214 unmarked TKO. Total hard TKOs: 0; soft TKOs: 0. Reply: mc_res_ok
(Note the 11214 port, that is one one used by the codfw proxies)
It is not a big deal at the moment, but we should figure out if we are currently reaching some limit (like too many requests at the same time, etc..). It might be the case that more codfw proxies are needed, or simply the proxy's available connections slots increased.