Page MenuHomePhabricator

Deadlock found when trying to get lock in RecentChangesUpdateJob::updateActiveUsersError
Open, Needs TriagePublic

Description

Error

MediaWiki version: 1.35.0-wmf.35

message
A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: DELETE FROM `querycachetwo` WHERE qcc_type = 'activeusers' AND (qcc_value < 1589219852)
Function: RecentChangesUpdateJob::updateActiveUsers
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.104)
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1663): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1640): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(1215): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/Database.php(3325): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->delete(string, string, string)
#5 /srv/mediawiki/php-1.35.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(518): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.35.0-wmf.35/includes/jobqueue/jobs/RecentChangesUpdateJob.php(240): Wikimedia\Rdbms\DBConnRef->delete(string, array, string)
#7 /srv/mediawiki/php-1.35.0-wmf.35/includes/jobqueue/jobs/RecentChangesUpdateJob.php(64): RecentChangesUpdateJob->updateActiveUsers()
#8 /srv/mediawiki/php-1.35.0-wmf.35/extensions/EventBus/includes/JobExecutor.php(79): RecentChangesUpdateJob->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#10 {main}

Impact

Statistics don't get updated.

Notes

There seems to be lock contention around trying to update the stats for Special:ActiveUsers. Seen on multipoe wikis: enwiki, commonswiki, dewiki, etc.

Details

Request ID
11f51aa4-36e2-4151-b84a-556d41199c10

Event Timeline

Krinkle created this task.Jun 10 2020, 6:39 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptJun 10 2020, 6:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr added a subscriber: Tgr.Fri, Jul 24, 10:38 AM

That would mean something else held a lock on querycachetwo and when RecentChangesUpdateJob::updateActiveUsers tried to acquire it, it turned out that it holds a lock that other thing is waiting for, right?
(Related: T198755: Log the query that caused a lock timeout)

That seems to be the only part of the code that touches querycachetwo, and it uses Database::lock to ensure that the critical section is not run multiple times in parallel. I guess this delete command (added in rMWbf8c90538dff: Make RecentChangesUpdateJob::updateActiveUsers more robust) should be inside the lock?