Page MenuHomePhabricator
Create Task
Maniphest T293859

Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query ... while transaction status is ERROR (after PHP timeout)
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from MediaWiki\Extension\AbuseFilter\Watcher\UpdateHitCountWatcher::updateHitCounts while transaction status is ERROR
A connection error occurred during a query. 
Query: SELECT  actor_user,actor_name,actor_id  FROM `actor`    WHERE actor_name = 'XXXXXX'  LIMIT 1  
Function: MediaWiki\User\ActorStore::findActorIdInternal
Error: 2006 MySQL server has gone away (db1138)
exception.trace
from /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(1619)
#0 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(1320): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed(string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(2654): Wikimedia\Rdbms\Database->query(string, string, integer)
#2 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#3 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(386): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#4 /srv/mediawiki/php-1.38.0-wmf.4/extensions/AbuseFilter/includes/Watcher/UpdateHitCountWatcher.php(60): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#5 /srv/mediawiki/php-1.38.0-wmf.4/extensions/AbuseFilter/includes/Watcher/UpdateHitCountWatcher.php(41): MediaWiki\Extension\AbuseFilter\Watcher\UpdateHitCountWatcher->updateHitCounts(Wikimedia\Rdbms\DBConnRef, array)
#6 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/MWCallableUpdate.php(38): MediaWiki\Extension\AbuseFilter\Watcher\UpdateHitCountWatcher->MediaWiki\Extension\AbuseFilter\Watcher\{closure}()
#7 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdates.php(515): MWCallableUpdate->doUpdate()
#8 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdates.php(391): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#9 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#10 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#11 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#12 /srv/mediawiki/php-1.38.0-wmf.4/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#13 /srv/mediawiki/php-1.38.0-wmf.4/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates()
#14 /srv/mediawiki/rpc/RunSingleJob.php(93): MediaWiki->restInPeace()
#15 {main}
exception.previous.trace
from /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(1807)
#0 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(1423): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(1323): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#2 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(2012): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/Database.php(2112): Wikimedia\Rdbms\Database->select(string, array, array, string, array, array)
#4 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(string, array, array, string, array)
#5 /srv/mediawiki/php-1.38.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(342): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.38.0-wmf.4/includes/user/ActorStore.php(381): Wikimedia\Rdbms\DBConnRef->selectRow(string, array, array, string, array)
#7 /srv/mediawiki/php-1.38.0-wmf.4/includes/user/ActorStore.php(412): MediaWiki\User\ActorStore->findActorIdInternal(string, Wikimedia\Rdbms\MaintainableDBConnRef)
#8 /srv/mediawiki/php-1.38.0-wmf.4/includes/filerepo/file/LocalFile.php(1899): MediaWiki\User\ActorStore->acquireActorId(User, Wikimedia\Rdbms\MaintainableDBConnRef)
#9 /srv/mediawiki/php-1.38.0-wmf.4/includes/filerepo/file/LocalFile.php(1829): LocalFile->recordUpload3(string, string, string, User, array, string, array, boolean, boolean)
#10 /srv/mediawiki/php-1.38.0-wmf.4/includes/upload/UploadBase.php(955): LocalFile->upload(string, string, string, integer, array, boolean, User, array)
#11 /srv/mediawiki/php-1.38.0-wmf.4/includes/jobqueue/jobs/PublishStashedFileJob.php(88): UploadBase->performUpload(string, string, boolean, User, array, NULL)
#12 /srv/mediawiki/php-1.38.0-wmf.4/extensions/EventBus/includes/JobExecutor.php(79): PublishStashedFileJob->run()
#13 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#14 {main}
Impact
Notes

A single occurrence in production. I don't know whether "MySQL server has gone away" error must be handled there or whether the deferred updates should be retried after X time.

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php
SubjectRepoBranchLines +/-
rdbms: directly use executeQuery() in mysql open() methodmediawiki/coremaster+9 -6
rdbms: add mysql timeout error codes to wasKnownStatementRollbackError()mediawiki/coremaster+5 -1
rdbms: make automatic connection recovery apply to more casesmediawiki/coremaster+901 -218
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Oct 20 2021, 8:00 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2021, 8:00 AM
Daimona added projects: MediaWiki-libs-Rdbms, MediaWiki-File-management.Oct 20 2021, 8:38 AM
Daimona subscribed.

This looks like a classic MySQL timeout, which could be caused by several things, and happens to have AF in its stack trace. I'm adding a couple of tags hoping that somebody more familiar with the related areas has some ideas about this, but if it's just a timeout this can probably be closed.

Restricted Application added a project: Performance-Team. · View Herald TranscriptOct 20 2021, 8:38 AM
Maintenance_bot added a project: Commons.Oct 20 2021, 8:46 AM
cjming moved this task from Untriaged to Oct 2021 on the Wikimedia-production-error board.Oct 20 2021, 6:54 PM
Krinkle assigned this task to aaron.Nov 22 2021, 7:54 PM
Krinkle moved this task from Inbox, needs triage to Doing: Prio Interrupt on the Performance-Team board.
Krinkle removed projects: MediaWiki-File-management, AbuseFilter.
Krinkle subscribed.

Seems like the timeout should not leave the transaction in an error state. Might be a state bug in rdbms, or rdbms being called incorrectly by ActorStore or DeferredUpdates.

Krinkle renamed this task from Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from MediaWiki\Extension\AbuseFilter\Watcher\UpdateHitCountWatcher::updateHitCounts while transaction status is ERROR to Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query ... while transaction status is ERROR (after PHP timeout).Nov 22 2021, 7:57 PM
gerritbot added a comment.Nov 24 2021, 12:44 AM

Change 737824 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] [WIP] rdbms: make automatic connection recovery apply to more cases

https://gerrit.wikimedia.org/r/737824

gerritbot added a project: Patch-For-Review.Nov 24 2021, 12:44 AM
Krinkle added a subtask: T297424: Reduce LBFactory::rollbackPrimaryChanges() callers in core.Dec 13 2021, 7:36 PM
aaron triaged this task as Medium priority.Jan 7 2022, 1:22 AM
Krinkle moved this task from Untriaged to Rdbms library on the MediaWiki-libs-Rdbms board.Jan 26 2022, 8:16 PM
gerritbot added a comment.Mar 10 2022, 12:08 AM

Change 737824 merged by jenkins-bot:

[mediawiki/core@master] rdbms: make automatic connection recovery apply to more cases

https://gerrit.wikimedia.org/r/737824

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2022, 12:10 AM
aaron closed subtask T297424: Reduce LBFactory::rollbackPrimaryChanges() callers in core as Resolved.Mar 15 2022, 8:20 PM
gerritbot added a comment.Mar 23 2022, 1:14 AM

Change 771755 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: add mysql timeout error codes to wasKnownStatementRollbackError()

https://gerrit.wikimedia.org/r/771755

gerritbot added a project: Patch-For-Review.Mar 23 2022, 1:14 AM
gerritbot added a comment.Mar 23 2022, 1:31 AM

Change 771755 merged by jenkins-bot:

[mediawiki/core@master] rdbms: add mysql timeout error codes to wasKnownStatementRollbackError()

https://gerrit.wikimedia.org/r/771755

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.5; 2022-03-28).Mar 23 2022, 2:00 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 23 2022, 2:10 AM
gerritbot added a comment.Mar 23 2022, 10:01 PM

Change 773352 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: directly use executeQuery() in mysql open() method

https://gerrit.wikimedia.org/r/773352

gerritbot added a project: Patch-For-Review.Mar 23 2022, 10:01 PM
gerritbot added a comment.Mar 29 2022, 11:09 PM

Change 773352 merged by jenkins-bot:

[mediawiki/core@master] rdbms: directly use executeQuery() in mysql open() method

https://gerrit.wikimedia.org/r/773352

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2022, 11:11 PM
ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.6; 2022-04-04); removed MW-1.39-notes (1.39.0-wmf.5; 2022-03-28).Mar 30 2022, 12:00 AM
Krinkle mentioned this in T305016: Think about rdbms reconnection logic.Mar 30 2022, 12:29 AM
Krinkle added a comment.Apr 6 2022, 1:03 AM

Logstash mediawiki-warnings query:

message:"while transaction status is ERROR" AND "MySQL server has gone away"

Still seen, but not since March 27. Will check again next week when wmf.6 has been on all wikis for a bit.

Umherirrender mentioned this in T277150: Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR.Apr 13 2022, 6:32 PM
Umherirrender mentioned this in T282726: Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR.
aaron closed this task as Resolved.Apr 28 2022, 5:24 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL