Page MenuHomePhabricator

mw1150 spams "memcached error for key" since May 29 3:00am UTC
Closed, ResolvedPublic

Description

mw1150 spams "memcached error for key" since May 29 3:00am UTC

https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors

Seems nutcracker is wild.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, Joe.

All servers have been ejected around 3 AM UTC and never recovered. We can probably monitor this kind of problems, and maybe also try to pin down a bit better what is the root cause of this - which is basically the same problem we had at T88730

Joe claimed this task.
Joe set Security to None.

I'm going to work on getting log event rates into graphite with the hope of using that to set some general "go look at logstash" alerts (T100735) for things like this.

demon triaged this task as Medium priority.Jul 9 2015, 4:39 PM