Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Krinkle
	Apr 21 2021, 6:22 PM

Description

Error

MediaWiki version: n/a

message

Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed in /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php:1011

trace

…
#3 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(737): Wikimedia\Rdbms\Database->ping(NULL)
#4 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(4833): Wikimedia\Rdbms\Database->pingAndCalculateLastTrxApplyTime()
#5 /srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/database/Database.php(960): Wikimedia\Rdbms\Database->rol

(Trimmed by an intermediary layer, not sure why)

Impact

Notes

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

Krinkle created this task.Apr 21 2021, 6:22 PM

Restricted Application added a project: Platform Engineering. · View Herald TranscriptApr 21 2021, 6:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Krinkle renamed this task from Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already to Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closed.Apr 21 2021, 6:23 PM

Krinkle added a project: Release-Engineering-Team (Logspam).Apr 21 2021, 6:59 PM

Krinkle moved this task from Untriaged to Apr 2021 on the Wikimedia-production-error board.

Krinkle removed a project: Release-Engineering-Team (Logspam).Apr 27 2021, 5:11 PM

Big spike of these around 10:40 UTC today. More than 300. Most requests coming in via api.php, but I also saw some via index.php

• AMooney triaged this task as High priority.Apr 27 2021, 9:23 PM

• AMooney edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.

• AMooney moved this task from Inbox to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.

Server admin log says:

10:40 volans@cumin1001: dbctl commit (dc=all): 'S4 RO, outage', diff saved to https://phabricator.wikimedia.org/P15585 and previous config saved to /var/cache/conftool/dbconfig/20210427-104057-volans.json

So apparently, s4 (commons) had problems. The errors in logstash come from other clusters, but perhaps these request are trying to talk to commons.

@daniel This task is about DBUnexpectedError: DB connection was already closed which I suspect is a logic error in the code.

It is indeed likely that this logic error is only exposed when some form of of db problem is active during the same time, but, unless the error message is wrongly phrased and/or if there is no added impact, I think this is not really about what circumstances may lead to it being triggered at one point or another, but rather about the secondary damages as result of this unexpected error.

Krinkle closed this task as Declined.Oct 11 2021, 2:18 AM

Restricted Application added a project: Performance-Team. · View Herald TranscriptOct 11 2021, 2:18 AM

Uncaught Wikimedia\Rdbms\DBUnexpectedError: DB connection was already closedClosed, DeclinedPublicPRODUCTION ERRORActions