Change Details

Both the memcached dashboards have a new trending error: https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious > Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED This exact error for that exact key is being reported about a 200 times per minute (of the 220 events per minute in total). It seems equally distributed among the different mw hosts. It started right after midnight, 2015-06-18 00:00 UTC.

Both the memcached dashboards have a new trending error: https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious > Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED > Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": SERVER ERROR Two two errors for that key only together make up 1890 errors per 5 minutes (out of ~ 1930 error events from memcached in total). It seems equally distributed among the different mw hosts. It started right after midnight, 2015-06-18 00:00 UTC.

Both the memcached dashboards have a new trending error: https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious > Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED > Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": SERVER ERROR This exactTwo two errors for that exacat key is being reported about a 200 timeonly together make up 1890 errors per 5 minute (of the 220s (out of ~ 1930 error events per minutefrom memcached in total). It seems equally distributed among the different mw hosts. It started right after midnight, 2015-06-18 00:00 UTC.