Page MenuHomePhabricator
Create Task
Maniphest T147866

Warning: Failed connecting to redis server at rdb100X.eqiad.wmnet: Connection timed out in /srv/mediawiki/php-1.28.0-wmf.XX/includes/libs/redis/RedisConnectionPool.php on line 235
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Description

Here's what logstash looks like at the time of writing:

Selection_140.png (345×1 px, 58 KB)

That's a ton of noise about redis timeouts. Way too much. The pairs of 473, 32, and 21 are from this (it emits two error logs for each timeout). That's 1052 errors in the past hour, just in the first page of fatals (there's one more in the next page of fatals).

All of the form:

Warning: Failed connecting to redis server at {redis_server}.eqiad.wmnet: Connection timed out in /srv/mediawiki/php-1.28.0-wmf.{XX}/includes/libs/redis/RedisConnectionPool.php on line 235

Event Timeline

greg created this task.Oct 11 2016, 5:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2016, 5:24 PM
greg added a subscriber: elukey.Oct 12 2016, 10:06 PM

@elukey do you have any idea on what we should here/other people to poke?

elukey added subscribers: aaron, Joe.Oct 13 2016, 9:44 AM

@greg probably @aaron might be able to help in figuring out the next steps. I am far from being an expert on this subject but what I'd really like to have is another log indicating what request triggered the timeouts, because it could help in figuring out if the issue is specific to a subset of them or not.

We could also check Redis service metrics but I am not sure if it will lead to anything useful (will do it as soon as possible if nobody else will do it).

@Joe might also have already investigated the issue coming up with a good explanation.

elukey closed this task as a duplicate of T125735: Warning: timed out after 0.2 seconds when connecting to rdb1001.eqiad.wmnet [110]: Connection timed out.Oct 17 2016, 1:03 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL