Page MenuHomePhabricator

Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed
Open, HighPublicPRODUCTION ERROR

Description

Error

MediaWiki version: n/a

message
Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed in /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php:1011
trace
…
#3 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(737): Wikimedia\Rdbms\Database->ping(NULL)
#4 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(4833): Wikimedia\Rdbms\Database->pingAndCalculateLastTrxApplyTime()
#5 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(960): Wikimedia\Rdbms\Database->rol

(Trimmed by an intermediary layer, not sure why)

Impact

Notes

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle renamed this task from Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already to Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed.Apr 21 2021, 6:23 PM

Big spike of these around 10:40 UTC today. More than 300. Most requests coming in via api.php, but I also saw some via index.php

Server admin log says:

10:40 volans@cumin1001: dbctl commit (dc=all): 'S4 RO, outage', diff saved to https://phabricator.wikimedia.org/P15585 and previous config saved to /var/cache/conftool/dbconfig/20210427-104057-volans.json

So apparently, s4 (commons) had problems. The errors in logstash come from other clusters, but perhaps these request are trying to talk to commons.

@daniel This task is about DBUnexpectedError: DB connection was already closed which I suspect is a logic error in the code.

It is indeed likely that this logic error is only exposed when some form of of db problem is active during the same time, but, unless the error message is wrongly phrased and/or if there is no added impact, I think this is not really about what circumstances may lead to it being triggered at one point or another, but rather about the secondary damages as result of this unexpected error.