Page MenuHomePhabricator

Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED
Closed, ResolvedPublic

Description

Both the memcached dashboards have a new trending error:
https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors
https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious

Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED
Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": SERVER ERROR

Two two errors for that key only together make up 1890 errors per 5 minutes (out of ~ 1930 error events from memcached in total).

It seems equally distributed among the different mw hosts.

It started right after midnight, 2015-06-18 00:00 UTC.

Event Timeline

Krinkle created this task.Jun 18 2015, 6:39 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2015, 6:39 AM

Given the timestamp, I suspect it's related to todays Main Page. And indeed that image is on the Main Page. Looking for the first two characters of the hash, "254363f3" -> "2/25/", I found:

<img src=".. upload.wikimedia.org/wikipedia/commons/thumb/2/25/Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg/300px-Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg" width="300" height="432" ..

https://en.wikipedia.org/wiki/File:Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg
https://commons.wikimedia.org/wiki/File:Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg

Krinkle updated the task description. (Show Details)Jun 18 2015, 6:46 AM
Krinkle set Security to None.
aaron added a subscriber: aaron.Jun 18 2015, 7:59 PM
aaron added a subscriber: BBlack.Jun 18 2015, 9:11 PM
aaron added a comment.Jun 19 2015, 9:01 PM

Note that since nutcracker is just a proxy, it could still indicate a problem on an mc host (the one that the key maps to). Not sure what logs we have there.

demon triaged this task as High priority.Jul 9 2015, 4:31 PM
demon added a subscriber: demon.

Marking high because mentioned task is also high.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 9 2015, 4:31 PM
fgiunchedi closed this task as Resolved.Dec 2 2015, 10:55 AM
fgiunchedi claimed this task.
fgiunchedi added a subscriber: fgiunchedi.

looks like this was fixed, or supplanted by different errors from https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached tentatively resolving