Page MenuHomePhabricator
Create Task
Maniphest T102916

Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED
Closed, ResolvedPublic
Actions

Description

Both the memcached dashboards have a new trending error:
https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-errors
https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious

Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": A TIMEOUT OCCURRED
Memcached error for key "WANCache:v:enwiki:image_redirect:254363f3d14af58bbe12c644ee69ccf7" on server "/var/run/nutcracker/nutcracker.sock:0": SERVER ERROR

Two two errors for that key only together make up 1890 errors per 5 minutes (out of ~ 1930 error events from memcached in total).

It seems equally distributed among the different mw hosts.

It started right after midnight, 2015-06-18 00:00 UTC.

Related Objects

Event Timeline

Krinkle created this task.Jun 18 2015, 6:39 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added projects: acl*sre-team, Wikimedia-production-error.
Krinkle subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2015, 6:39 AM
Krinkle added a comment.Jun 18 2015, 6:43 AM

Given the timestamp, I suspect it's related to todays Main Page. And indeed that image is on the Main Page. Looking for the first two characters of the hash, "254363f3" -> "2/25/", I found:

<img src=".. upload.wikimedia.org/wikipedia/commons/thumb/2/25/Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg/300px-Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg" width="300" height="432" ..

https://en.wikipedia.org/wiki/File:Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg
https://commons.wikimedia.org/wiki/File:Grand_Duchess_Anastasia_Nikolaevna_Crisco_edit_letters_removed.jpg

Krinkle updated the task description. (Show Details)Jun 18 2015, 6:46 AM
Krinkle set Security to None.
aaron subscribed.Jun 18 2015, 7:59 PM
aaron added a subscriber: BBlack.Jun 18 2015, 9:11 PM
aaron added a comment.Jun 19 2015, 9:01 PM

Note that since nutcracker is just a proxy, it could still indicate a problem on an mc host (the one that the key maps to). Not sure what logs we have there.

aaron mentioned this in T104225: enwiki Main_Page timeouts.Jun 29 2015, 7:36 PM
• demon triaged this task as High priority.Jul 9 2015, 4:31 PM
• demon subscribed.

Marking high because mentioned task is also high.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 9 2015, 4:31 PM
• demon moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Jul 9 2015, 4:31 PM
Nemo_bis mentioned this in T69817: Monitor for anomalies/spikes in read failures of memcached.Jul 17 2015, 6:57 AM
Nemo_bis added projects: Sustainability, Performance Issue.Jul 17 2015, 7:36 AM
Nemo_bis subscribed.
fgiunchedi closed this task as Resolved.Dec 2 2015, 10:55 AM
fgiunchedi claimed this task.
fgiunchedi subscribed.

looks like this was fixed, or supplanted by different errors from https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached tentatively resolving

• demon moved this task from Dec2019/1.35.wmf.10+ to Resolved on the Wikimedia-production-error board.Dec 3 2015, 7:11 PM
Krinkle mentioned this in T121187: Memcached error "SERVER ERROR" from nutcracker .Dec 11 2015, 2:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits