Page MenuHomePhabricator
Create Task
Maniphest T277150

Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.36.0-wmf.33

message
Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR
exception.trace
from /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1505)
#0 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1224): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed(string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/DatabaseMysqlBase.php(1243): Wikimedia\Rdbms\Database->query(string, string, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(5299): Wikimedia\Rdbms\DatabaseMysqlBase->unlock(string, string)
#3 [internal function]: Wikimedia\Rdbms\Database->Wikimedia\Rdbms\{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#4 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(4141): call_user_func(Closure, integer, Wikimedia\Rdbms\DatabaseMysqli)
#5 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1871): Wikimedia\Rdbms\Database->runOnTransactionIdleCallbacks(integer)
#6 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2255): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#7 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1875): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#8 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(346): Wikimedia\Rdbms\LoadBalancer->runMasterTransactionIdleCallbacks(string, integer)
#9 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(236): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer)
#10 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(348): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure)
#11 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(333): Wikimedia\Rdbms\LBFactory->executePostTransactionCallbacks()
#12 /srv/mediawiki/php-1.36.0-wmf.33/includes/exception/MWExceptionHandler.php(135): Wikimedia\Rdbms\LBFactory->rollbackMasterChanges(string)
#13 /srv/mediawiki/php-1.36.0-wmf.33/extensions/EventBus/includes/JobExecutor.php(112): MWExceptionHandler::rollbackMasterChangesAndLog(Wikimedia\Rdbms\DBQueryDisconnectedError)
#14 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#15 {main}
exception.previous.trace
from /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1701)
#0 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1327): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(4854): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1755): Wikimedia\Rdbms\Database->ping()
#4 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2255): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#5 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1761): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#6 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(247): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#7 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(236): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(249): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#9 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/lbfactory/LBFactory.php(308): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#10 /srv/mediawiki/php-1.36.0-wmf.33/extensions/EventBus/includes/JobExecutor.php(287): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string, array)
#11 /srv/mediawiki/php-1.36.0-wmf.33/extensions/EventBus/includes/JobExecutor.php(81): MediaWiki\Extension\EventBus\JobExecutor->commitMasterChanges(Wikimedia\Rdbms\LBFactoryMulti, string)
#12 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#13 {main}

Impact

Only a few errors here and there. It seems to be solely that CategoryMembershipChangeJob job.

Notes

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
rdbms: make unlock() no-op for locks that are not heldAaron Schulzmediawiki/coremaster+15 -6
rdbms: improve reconnection handling after Database::getScopedLockAndFlush()Aaron Schulzmediawiki/coremaster+36 -3
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Mar 11 2021, 10:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2021, 10:36 AM
hashar added a project: Platform Engineering.Mar 11 2021, 10:38 AM
phuedx subscribed.Mar 11 2021, 10:42 AM

This won't fix the issue but Wikimedia\Rdbms\Database::query should at least be documented as throwing DBTransactionStateError.

gerritbot added a comment.Mar 11 2021, 5:17 PM

Change 670890 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/core@master] Document missing exception for IDatabase::query()

https://gerrit.wikimedia.org/r/670890

gerritbot added a project: Patch-For-Review.Mar 11 2021, 5:17 PM
• AMooney triaged this task as Medium priority.Mar 16 2021, 2:14 PM
• AMooney edited projects, added: Platform Team Workboards (Clinic Duty Team); removed: Platform Engineering.
• AMooney moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.
Krinkle moved this task from Untriaged to Mar 2021 on the Wikimedia-production-error board.Mar 27 2021, 3:05 AM
hashar unsubscribed.Mar 29 2021, 12:37 PM
Krinkle removed a project: Patch-For-Review.May 13 2021, 8:20 PM
Umherirrender mentioned this in T282726: Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR.Aug 19 2021, 8:28 PM
Umherirrender subscribed.Apr 13 2022, 6:32 PM

Please include the exception message of the previous exception to be able to distinct the "while transaction status is ERROR" tasks.

Is the previous exception about lost connection ("MySQL server has gone away")? That sounds like T293859

Umherirrender unsubscribed.Jun 20 2022, 4:17 PM
Krinkle updated the task description. (Show Details)Nov 26 2022, 7:57 AM
Krinkle edited Stack Trace. (Show Details)
Krinkle edited projects, added: MediaWiki-Recent-changes; removed: MediaWiki-General.Nov 26 2022, 8:03 AM
Krinkle subscribed.

Still happening ~700 times a week.

A recent sample:

exception
DBQueryDisconnectedError: Cannot execute query from CategoryMembershipChangeJob::run while session status is ERROR

#0 libs/rdbms/Database.php(1330): TransactionManager->assertSessionStatus()
#1 libs/rdbms/Database.php(879): Database->assertQueryIsCurrentlyAllowed(string, string)
#2 libs/rdbms/DatabaseMysqlBase.php(948): Database->query(string, string, integer)
..
#4 libs/rdbms/Database.php(3400): Database->unlock(string, string)
..
#6 libs/rdbms/LoadBalancer.php(1744): Database->runOnTransactionIdleCallbacks(integer, array)
..
#9 MWExceptionHandler.php(167): LBFactory->rollbackPrimaryChanges(string)
..
#11 extensions/EventBus/includes/JobExecutor.php(112): MWExceptionHandler::rollbackPrimaryChangesAndLog(DBQueryDisconnectedError)
#12 MediaWiki\Extension\EventBus\JobExecutor->execute()
exception.previous
A connection error occurred during a query. 
Query: SELECT 1 AS ping
Function: Wikimedia\Rdbms\Database::ping
Error: 2006 MySQL server has gone away

#0 includes/libs/rdbms/database/Database.php(1178): Database->getQueryException()
#1 includes/libs/rdbms/database/Database.php(1046): Database->attemptQuery()
#2 includes/libs/rdbms/database/Database.php(883): Database->executeQuery()
#3 includes/libs/rdbms/database/Database.php(2959): Database->query()
#4 includes/libs/rdbms/loadbalancer/LoadBalancer.php(1640): Database->ping()
#5 includes/libs/rdbms/lbfactory/LBFactory.php(402): LoadBalancer->approvePrimaryChanges()
#6 extensions/EventBus/includes/JobExecutor.php(287): LBFactory->commitPrimaryChanges()
#7 extensions/EventBus/includes/JobExecutor.php(81): JobExecutor->commitPrimaryChanges()
#8 MediaWiki\Extension\EventBus\JobExecutor->execute()
Restricted Application added a project: Growth-Team. · View Herald TranscriptNov 26 2022, 8:03 AM
Krinkle added a project: Performance-Team.Nov 26 2022, 8:03 AM
Krinkle assigned this task to aaron.Nov 28 2022, 7:50 PM
• larissagaulia moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.Nov 28 2022, 7:50 PM
kostajh moved this task from Inbox to Triaged on the Growth-Team board.Dec 7 2022, 12:27 PM
aaron merged a task: T282726: Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from CategoryMembershipChangeJob::run while transaction status is ERROR.Feb 13 2023, 7:52 PM
aaron added subscribers: thcipriani, kostajh, MMiller_WMF, Tgr.
Krinkle added a comment.Feb 13 2023, 8:18 PM
In T328043#8576807, @Krinkle wrote:
In T328043#8576032, @aaron wrote:

I don't see a problem with selectDomain() itself. It looks like a query in EchoUserLocator::locateArticleCreator was really slow and triggered an excimer timeout.

This happened within DeferredUpdates, which tries to catch errors and move on to other updates, […]. The main point of the excimer timeouts is to limit run time, give a chance to log, and maybe give a clean error to the client. Catching/ignoring them undermines some of that and runs the risk of hitting uglier fpm/apache timeouts.

I think if the main response portion, pre-send update, or earlier post-send update times out, we should still let try to let other updates run, especially ones that might run hooks and/or queue jobs. However, I could be convinced otherwise if that's the only benefit left.

As it stands though, I lean toward not changing it because it seems orthogonal to the observed problem.

  • Even if we stop forego most updates after a timeout in the main portion, there may still be a trivial Rdbms query triggered by the skin or as part of processing a late hook or Job creation that needs some basic data for its parameters. This data is typically cachable in Memcached, but can just as well cache-miss.
  • The problem here seems first and foremost that the Rdbms object is corrupting its state and thus not fulfilling its contract. As part of a related production error task we developed (not insignificant) complexity in the form of RequestTimeout and critical sections, to prevent this. Do we want to make that actually work or should we consider removing that if it isn't working? If remove it, how do we mitigate the production error?
In T328043#8581166, @aaron wrote:

The advantage of excimer timers is that they can happen at anytime in PHP code. It's tricky to avoid corrupting the state of objects though. We really should just be cleaning up and shutting down. If we want to handle it more gracefully in Database, one could:

  • Rely on the fact that Zend C wrapper methods do not get preempted by excimer. DatabaseMysli::doSingleStatementQuery could catch the error, stage it, and have it rethrown (this gets more complicated with nested critical sections and/or when a section include several queries). It seems too messy.
  • DeferredUpdates could use some LBFactory method to discard all DB connections when a timeout exceptions are caught.
  • MWExceptionHandler::rollbackPrimaryChanges could discard such DB connections.

It could also help to:

  • set max_statement_time on the session level (different for web vs job vs cli).
  • set MYSQLI_OPT_READ_TIMEOUT for regular DB connections (different for web vs job vs cli). This would match the idea in CommonSettings.php for $wgHTTPMaxTimeout /$wgHTTPMaxConnectTimeout and $wgRequestTimeLimit;
In T328043#8612103, @Krinkle wrote:

MWExceptionHandler::rollbackPrimaryChanges could discard such DB connections.

Per todays triage meeting, let's go with this one.

Krinkle raised the priority of this task from Medium to High.Feb 13 2023, 8:18 PM
Krinkle moved this task from To-do: Goals prioritized current Quarter to Doing: Prio Interrupt on the Performance-Team board.
aaron merged a task: T328043: Wikimedia\Rdbms\DBUnexpectedError: Cannot execute Wikimedia\Rdbms\Database::selectDomain critical section while session state is out of sync..Feb 13 2023, 8:19 PM
aaron added subscribers: brennen, aaron.
gerritbot added a comment.Apr 5 2023, 4:07 AM

Change 905778 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: improve reconnection handling after Database::getScopedLockAndFlush()

https://gerrit.wikimedia.org/r/905778

gerritbot added a project: Patch-For-Review.Apr 5 2023, 4:07 AM

Change 905779 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: make unlock() no-op for locks that are not held

https://gerrit.wikimedia.org/r/905779

gerritbot added a comment.Apr 5 2023, 10:50 PM

Change 905778 merged by jenkins-bot:

[mediawiki/core@master] rdbms: improve reconnection handling after Database::getScopedLockAndFlush()

https://gerrit.wikimedia.org/r/905778

gerritbot added a comment.Apr 5 2023, 10:52 PM

Change 905779 merged by jenkins-bot:

[mediawiki/core@master] rdbms: make unlock() no-op for locks that are not held

https://gerrit.wikimedia.org/r/905779

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.4; 2023-04-10).Apr 5 2023, 11:00 PM
Krinkle closed this task as Resolved.Apr 5 2023, 11:36 PM
Krinkle removed a project: Platform Team Workboards (Clinic Duty Team).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits