Page MenuHomePhabricator

Spamming DB transaction related issues
Closed, ResolvedPublic

Description

Since upgrading to MediaWiki 1.32 i have seen a lot of these in the logs:

[11-Jan-2019 18:32:12 UTC] PHP Notice:  DB transaction writes or callbacks still pending (User::saveSettings). in /srv/mediawiki/w/includes/libs/rdbms/database/Database.php on line 4714
PHP Fatal error:  Uncaught RuntimeException: Transaction callbacks are still pending:
User::saveSettings in /srv/mediawiki/w/includes/libs/rdbms/database/Database.php:977
Stack trace:
#0 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1279): Wikimedia\Rdbms\Database->close()
#1 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1774): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Object(Wikimedia\Rdbms\DatabaseMysqli))
#2 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1280): Wikimedia\Rdbms\LoadBalancer->forEachOpenConnection(Object(Closure))
#3 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1269): Wikimedia\Rdbms\LoadBalancer->closeAll()
#4 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1992): Wikimedia\Rdbms\LoadBalancer->disable()
#5 [internal function]: Wikimedia\Rdbms\LoadBalancer->__destruct()
#6 {main}
  thrown in /srv/mediawiki/w/includes/libs/rdbms/database/Database.php on line 977
PHP Fatal error:  Uncaught Wikimedia\Rdbms\DBTransactionError: Transaction round stage must be 'cursory' (not 'within-rollback') in /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php:703
Stack trace:
#0 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(251): Wikimedia\Rdbms\LBFactory->assertTransactionRoundStage('cursory')
#1 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(179): Wikimedia\Rdbms\LBFactory->commitMasterChanges('Wikimedia\\Rdbms...')
#2 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(152): Wikimedia\Rdbms\LBFactory->shutdown(0)
#3 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(728): Wikimedia\Rdbms\LBFactory->destroy()
#4 [internal function]: Wikimedia\Rdbms\LBFactory->__destruct()
#5 {main}
  thrown in /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php on line 703
/w/api.php   Wikimedia\Rdbms\DBTransactionStateError from line 1341 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Cannot execute query from Wikimedia\Rdbms\Database::ping while transaction status is ERROR.
#0 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(1127): Wikimedia\Rdbms\Database->assertTransactionStatus(string, string)
#1 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(4119): Wikimedia\Rdbms\Database->query(string, string, boolean)
#2 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(722): Wikimedia\Rdbms\Database->ping(NULL)
#3 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(3973): Wikimedia\Rdbms\Database->pendingWriteQueryDuration(string)
#4 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1541): Wikimedia\Rdbms\Database->rollback(string, string)
#5 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1786): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#6 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1542): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#7 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(219): Wikimedia\Rdbms\LoadBalancer->rollbackMasterChanges(string)
#8 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(425): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#9 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(221): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#10 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(289): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#11 /srv/mediawiki/w/includes/exception/MWExceptionHandler.php(123): Wikimedia\Rdbms\LBFactory->rollbackMasterChanges(string)
#12 /srv/mediawiki/w/includes/api/ApiMain.php(571): MWExceptionHandler::rollbackMasterChangesAndLog(MWException)
#13 /srv/mediawiki/w/includes/api/ApiMain.php(538): ApiMain->handleException(MWException)
#14 /srv/mediawiki/w/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/w/api.php(87): ApiMain->execute()
#16 {main}

It is very spammy (and not sure if it's impacting users)

Event Timeline

Paladox created this task.Jan 11 2019, 8:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2019, 8:27 PM
Paladox updated the task description. (Show Details)Jan 11 2019, 8:29 PM
Paladox updated the task description. (Show Details)Jan 11 2019, 8:32 PM

Change 483852 had a related patch set uploaded (by Paladox; owner: Aaron Schulz):
[mediawiki/core@REL1_32] rdbms: avoid transaction status errors from ping() in rollback()

https://gerrit.wikimedia.org/r/483852

Reedy renamed this task from DB transaction writes or callbacks still pending (User::saveSettings) to Spamming DB transaction related issues.

Change 483852 merged by jenkins-bot:
[mediawiki/core@REL1_32] rdbms: avoid transaction status errors from ping() in rollback()

https://gerrit.wikimedia.org/r/483852

https://gerrit.wikimedia.org/r/483852 seems to have fixed all the spammy transaction errors.

Paladox closed this task as Resolved.Jan 12 2019, 11:36 PM

Been monitoring for 24 hours and haven't seen any spammy transactions messages since i deployed the fix.