Page MenuHomePhabricator
Create Task
Maniphest T278734

mw1304: Memcached error for key X on server 127.0.0.1:11213: A TIMEOUT OCCURRED
Closed, ResolvedPublic
Actions

Description

Since roughly 3:40 am UTC, mw1304 emits TIMEOUT OCCURRED memcached errors at a rate of roughly 400 events per minute.

https://logstash.wikimedia.org/goto/925eb928dffdd7727e80eb95415feb76

mw1304_memcached.png (458×945 px, 63 KB)

Event Timeline

hashar created this task.Mar 30 2021, 7:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2021, 7:31 AM
hashar added a comment.Mar 30 2021, 7:32 AM

It also does not show up in https://grafana.wikimedia.org/d/000000377/host-overview so maybe the host is broken somehow.

Stashbot added a comment.Mar 30 2021, 7:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-30T07:37:21Z] <elukey> restart-php7.2-fpm on mw1304, jobrunner completely overwhelmed by ffmpeg/transcode jobs (not publishing metrics, erroring out for memcached timeouts) - T278734

hashar added a comment.Mar 30 2021, 7:41 AM

The timeout errors have vanished. No idea why the job runner would over run video transcoding on a given host though.

hashar closed this task as Resolved.Mar 30 2021, 12:45 PM
hashar assigned this task to elukey.

Root cause is not addressed but flushing the stuck php transcode jobs has made the server responsive again.

hashar added a comment.Mar 30 2021, 4:48 PM

The same issue happened later which is now tracked in an incident document.

RhinosF1 subscribed.Mar 30 2021, 4:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits