Page MenuHomePhabricator

"Unable to connect to redis server" log spam
Closed, DuplicatePublic

Description

Logstash shows up a good spam of messages "Unable to connect to redis server" on https://logstash.wikimedia.org/#/dashboard/elasticsearch/redis

The top offenders:

server:port24 hours hits
rdb1001.eqiad.wmnet:6380166685
rdb1001.eqiad.wmnet:6379165655
rdb1001.eqiad.wmnet:6381159330
rdb1005.eqiad.wmnet:638073329
rdb1005.eqiad.wmnet:638168856
rdb1005.eqiad.wmnet:637965288
rdb1003.eqiad.wmnet:637913038
rdb1007.eqiad.wmnet:637913015
rdb1007.eqiad.wmnet:638012778
rdb1007.eqiad.wmnet:638112351

Maybe it is due to the currently abnormal of refreshLink jobs being processed. Maybe the Nutcracker proxy is not used or redis instance is being overloaded / has not enough connections.

Event Timeline

hashar created this task.Mar 15 2016, 10:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2016, 10:21 PM
fgiunchedi triaged this task as Medium priority.Apr 27 2016, 10:16 AM
fgiunchedi added a subscriber: fgiunchedi.

also it seems load-related as there is a daily pattern for "unable to connect to redis server"