Over the last hour:
| Count | Message |
|---|---|
| 20939 | Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY |
| 316 | Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR |
And puppet reports:
Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Nutcracker/Service[nutcracker]: Unscheduling refresh on Service[nutcracker]
Mentioned in SAL [2016-02-26T11:24:32Z] <hashar> puppet keep restarting nutcracker apparently T128177
On deployment-mediawiki02
# /sbin/status nutcracker nutcracker stop/waiting # ps -A f|grep nutcr 19327 pts/2 S+ 0:00 \_ grep --color=auto nutcr 1256 ? Sl 29:58 /usr/sbin/nutcracker --verbose=5 --mbuf-size=65536 -d # cat /run/nutcracker/nutcracker.pid 1256
Which does not make sense. So I have stopped it, which got rid of the pidfile and starting it up again pass.
# /etc/init.d/nutcracker stop * Stopping nutcracker memcached and redis proxy [OK] # cat /run/nutcracker/nutcracker.pid cat: /run/nutcracker/nutcracker.pid: No such file or directory # /sbin/status nutcracker nutcracker stop/waiting # /sbin/start nutcracker nutcracker start/running, process 19409 # /sbin/status nutcracker nutcracker start/running, process 19409 # cat /run/nutcracker/nutcracker.pid 19409
I am blaming puppet / upstart for being lames.
Mentioned in SAL [2016-02-26T11:53:12Z] <hashar> memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) T128177
Mentioned in SAL [2016-02-26T11:53:54Z] <hashar> Restarted memcached on deployment-memc02 T128177
Root cause: memcached service on deployment-memc02 went wild. Restarting it has fixed the issue.