Page MenuHomePhabricator

Deadlock found when trying to get lock in RecentChangesUpdateJob::updateActiveUsersError
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

MediaWiki version: 1.35.0-wmf.35

message
A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: DELETE FROM `querycachetwo` WHERE qcc_type = 'activeusers' AND (qcc_value < 1589219852)
Function: RecentChangesUpdateJob::updateActiveUsers
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.104)
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1663): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1640): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1215): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(3325): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->delete(string, string, string)
#5 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(518): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.35.0-wmf.35/includes/jobqueue/jobs/RecentChangesUpdateJob.php(240): Wikimedia\Rdbms\DBConnRef->delete(string, array, string)
#7 /srv/mediawiki/php-1.35.0-wmf.35/includes/jobqueue/jobs/RecentChangesUpdateJob.php(64): RecentChangesUpdateJob->updateActiveUsers()
#8 /srv/mediawiki/php-1.35.0-wmf.35/extensions/EventBus/includes/JobExecutor.php(79): RecentChangesUpdateJob->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#10 {main}

Impact

Statistics don't get updated.

Notes

There seems to be lock contention around trying to update the stats for Special:ActiveUsers. Seen on multipoe wikis: enwiki, commonswiki, dewiki, etc.

Screenshot 2020-06-10 at 19.40.00.png (1×1 px, 170 KB)

Details

Request ID
11f51aa4-36e2-4151-b84a-556d41199c10

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

That would mean something else held a lock on querycachetwo and when RecentChangesUpdateJob::updateActiveUsers tried to acquire it, it turned out that it holds a lock that other thing is waiting for, right?
(Related: T198755: Log the query that caused a lock timeout)

That seems to be the only part of the code that touches querycachetwo, and it uses Database::lock to ensure that the critical section is not run multiple times in parallel. I guess this delete command (added in rMWbf8c90538dff: Make RecentChangesUpdateJob::updateActiveUsers more robust) should be inside the lock?

Change 676447 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] changes: Move unlock to cover DELETE query for activeusers in RC jobs

https://gerrit.wikimedia.org/r/676447

Still seen. Using a lock there makes sense to me.

Krinkle triaged this task as Medium priority.Apr 1 2021, 7:43 PM

Change 676447 merged by jenkins-bot:

[mediawiki/core@master] changes: Move unlock to cover DELETE query for activeusers in RC jobs

https://gerrit.wikimedia.org/r/676447