Page MenuHomePhabricator

Failed connecting to redis server at rdbXXX.eqiad.wmnet: Bad file descriptor in /srv/mediawiki/php-1.29.0-wmf.12/includes/libs/redis/RedisConnectionPool.php on line 235
Open, MediumPublic

Description

Logstash: https://logstash.wikimedia.org/goto/b008afebe545d2e29084acef88631416

We have a bunch of errors such as:

ErrorException from line 235 of /srv/mediawiki/php-1.29.0-wmf.12/includes/libs/redis/RedisConnectionPool.php:
PHP Warning: Failed connecting to redis server at rdb1001.eqiad.wmnet: Bad file descriptor

That surfaced since we had the log levels tweaked :)

First log entry was at 2017-02-21 01:21:42 UTC from SAL:

02:24	<l10nupdate@tin>	ResourceLoader cache refresh completed at Tue Feb 21 02:24:37 UTC 2017 (duration 5m 20s)
02:19	<l10nupdate@tin>	scap sync-l10n completed (1.29.0-wmf.12) (duration: 07m 20s)
01:17	<tstarling@tin>	Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 42s)
20:31	<gehel>	taking threaddumps and restarting elastic1017 (high load)

Tim Starling patch 9eafbebdd882a562171fd0e46dcd5a9ec1e55cf8 actually enables some more logging which is/was T45086

commit 9eafbebdd882a562171fd0e46dcd5a9ec1e55cf8
Author:     Tim Starling <tstarling@wikimedia.org>
AuthorDate: Mon Feb 20 19:03:01 2017
Commit:     Tim Starling <tstarling@wikimedia.org>
CommitDate: Tue Feb 21 00:54:55 2017

    Route PHP warnings from the handler into udp2log
    
    Don't send suppressed errors
    
    Bug: T45086
    Change-Id: If34503314d7f359c005d030329111be912a758d9

Event Timeline

hashar created this task.Feb 22 2017, 2:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2017, 2:29 PM
elukey added a subscriber: elukey.Apr 7 2017, 12:42 PM
elukey triaged this task as Medium priority.Apr 10 2017, 1:46 PM
elukey added a subscriber: tstarling.
Krinkle added a subscriber: Krinkle.
Sample from api.php
PHP Warning: Failed connecting to redis server at 10.64.0.80: Connection timed out

#1 [internal function]: Redis->doConnect(string, integer, integer, string, integer, boolean)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/redis/RedisConnectionPool.php(236): Redis->connect(string, integer, integer)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/RedisLockManager.php(247): RedisConnectionPool->getConnection(string, Monolog\Logger)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/QuorumLockManager.php(170): RedisLockManager->isServerUp(string)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/QuorumLockManager.php(135): QuorumLockManager->collectPledgeQuorum(integer, Closure$QuorumLockManager::doLockingRequestBucket;649)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/QuorumLockManager.php(66): QuorumLockManager->doLockingRequestBucket(integer, array)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/LockManager.php(133): QuorumLockManager->doLockByType(array)
#8 [internal function]: Closure$LockManager::lockByType()
#9 /srv/mediawiki/php-1.32.0-wmf.20/vendor/wikimedia/wait-condition-loop/src/WaitConditionLoop.php(92): call_user_func(Closure$LockManager::lockByType;651)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/LockManager.php(139): Wikimedia\WaitConditionLoop->invoke()
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/lockmanager/LockManager.php(116): LockManager->lockByType(array, integer)
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(1304): LockManager->lock(array, integer, integer)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(2221): FileBackend->lockFiles(array, integer, integer)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(2253): LocalFile->acquireFileLock()
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1349): LocalFile->lock()
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadBase.php(869): LocalFile->upload(string, string, string, integer, array, boolean, User, NULL)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(828): UploadBase->performUpload(string, string, boolean, User, NULL)
#18 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(146): ApiUpload->performUpload(array)
#19 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(104): ApiUpload->getContextResult()
#20 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(1587): ApiUpload->execute()
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM