Page MenuHomePhabricator
Create Task
Maniphest T374244

CI failure due to deadlock on cuci_user table
Closed, ResolvedPublic
Actions

Description

Seen in this gate-and-submit build for this Gerrit change:

mw-dberror.log:Fri Sep 6 13:54:45 UTC 2024	a6cfaed359e0	wikidb	Error 1213 from MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob::run, Deadlock found when trying to get lock; try restarting transaction INSERT INTO `cuci_user` (ciu_timestamp,ciu_ciwm_id,ciu_central_id) VALUES ('20240906135443',1,92) ON DUPLICATE KEY UPDATE ciu_timestamp = '20240906135443' localhost:/workspace/db/quibble-mysql-y0boxzq3/socket
mw-dberror.log:#0 /workspace/src/includes/libs/rdbms/database/Database.php(1151): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, int, string, string)
mw-dberror.log:#1 /workspace/src/includes/libs/rdbms/database/Database.php(643): Wikimedia\Rdbms\Database->reportQueryError(string, int, string, string, bool)
mw-dberror.log:#2 /workspace/src/includes/libs/rdbms/database/DatabaseMySQL.php(542): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
mw-dberror.log:#3 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\DatabaseMySQL->upsert(string, array, array, array, string)
mw-dberror.log:#4 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(567): Wikimedia\Rdbms\DBConnRef->__call(string, array)
mw-dberror.log:#5 /workspace/src/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\DBConnRef->upsert(string, array, array, array, string)
mw-dberror.log:#6 /workspace/src/extensions/CheckUser/src/Jobs/UpdateUserCentralIndexJob.php(60): Wikimedia\Rdbms\InsertQueryBuilder->execute()
mw-dberror.log:#7 /workspace/src/includes/jobqueue/JobRunner.php(369): MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob->run()
mw-dberror.log:#8 /workspace/src/includes/jobqueue/JobRunner.php(330): JobRunner->doExecuteJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-dberror.log:#9 /workspace/src/includes/jobqueue/JobRunner.php(225): JobRunner->executeJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-dberror.log:#10 /workspace/src/includes/specials/SpecialRunJobs.php(136): JobRunner->run(array)
mw-dberror.log:#11 /workspace/src/includes/specials/SpecialRunJobs.php(121): MediaWiki\Specials\SpecialRunJobs->doRun(array)
mw-dberror.log:#12 /workspace/src/includes/specialpage/SpecialPage.php(719): MediaWiki\Specials\SpecialRunJobs->execute(null)
mw-dberror.log:#13 /workspace/src/includes/specialpage/SpecialPageFactory.php(1708): MediaWiki\SpecialPage\SpecialPage->run(null)
mw-dberror.log:#14 /workspace/src/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
mw-dberror.log:#15 /workspace/src/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
mw-dberror.log:#16 /workspace/src/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
mw-dberror.log:#17 /workspace/src/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
mw-dberror.log:#18 {main}
mw-dberror.log:Fri Sep 6 13:59:10 UTC 2024	a6cfaed359e0	wikidb	Error 1213 from MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob::run, Deadlock found when trying to get lock; try restarting transaction INSERT INTO `cuci_user` (ciu_timestamp,ciu_ciwm_id,ciu_central_id) VALUES ('20240906135836',1,1086) ON DUPLICATE KEY UPDATE ciu_timestamp = '20240906135836' localhost:/workspace/db/quibble-mysql-y0boxzq3/socket
mw-dberror.log:#0 /workspace/src/includes/libs/rdbms/database/Database.php(1151): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, int, string, string)
mw-dberror.log:#1 /workspace/src/includes/libs/rdbms/database/Database.php(643): Wikimedia\Rdbms\Database->reportQueryError(string, int, string, string, bool)
mw-dberror.log:#2 /workspace/src/includes/libs/rdbms/database/DatabaseMySQL.php(542): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
mw-dberror.log:#3 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\DatabaseMySQL->upsert(string, array, array, array, string)
mw-dberror.log:#4 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(567): Wikimedia\Rdbms\DBConnRef->__call(string, array)
mw-dberror.log:#5 /workspace/src/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\DBConnRef->upsert(string, array, array, array, string)
mw-dberror.log:#6 /workspace/src/extensions/CheckUser/src/Jobs/UpdateUserCentralIndexJob.php(60): Wikimedia\Rdbms\InsertQueryBuilder->execute()
mw-dberror.log:#7 /workspace/src/includes/jobqueue/JobRunner.php(369): MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob->run()
mw-dberror.log:#8 /workspace/src/includes/jobqueue/JobRunner.php(330): JobRunner->doExecuteJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-dberror.log:#9 /workspace/src/includes/jobqueue/JobRunner.php(225): JobRunner->executeJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-dberror.log:#10 /workspace/src/includes/specials/SpecialRunJobs.php(136): JobRunner->run(array)
mw-dberror.log:#11 /workspace/src/includes/specials/SpecialRunJobs.php(121): MediaWiki\Specials\SpecialRunJobs->doRun(array)
mw-dberror.log:#12 /workspace/src/includes/specialpage/SpecialPage.php(719): MediaWiki\Specials\SpecialRunJobs->execute(null)
mw-dberror.log:#13 /workspace/src/includes/specialpage/SpecialPageFactory.php(1708): MediaWiki\SpecialPage\SpecialPage->run(null)
mw-dberror.log:#14 /workspace/src/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
mw-dberror.log:#15 /workspace/src/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
mw-dberror.log:#16 /workspace/src/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
mw-dberror.log:#17 /workspace/src/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
mw-dberror.log:#18 {main}
mw-error.log:2024-09-06 13:54:45 a6cfaed359e0 wikidb: [d53cbb1f3391b32aca6c1e40] /index.php   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction
mw-error.log:Function: MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob::run
mw-error.log:Query: INSERT INTO `cuci_user` (ciu_timestamp,ciu_ciwm_id,ciu_central_id) VALUES ('20240906135443',1,92) ON DUPLICATE KEY UPDATE ciu_timestamp = '20240906135443'
mw-error.log:#0 /workspace/src/includes/libs/rdbms/database/Database.php(1177): Wikimedia\Rdbms\Database->getQueryException(string, int, string, string)
mw-error.log:#1 /workspace/src/includes/libs/rdbms/database/Database.php(1151): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, int, string, string)
mw-error.log:#2 /workspace/src/includes/libs/rdbms/database/Database.php(643): Wikimedia\Rdbms\Database->reportQueryError(string, int, string, string, bool)
mw-error.log:#3 /workspace/src/includes/libs/rdbms/database/DatabaseMySQL.php(542): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
mw-error.log:#4 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\DatabaseMySQL->upsert(string, array, array, array, string)
mw-error.log:#5 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(567): Wikimedia\Rdbms\DBConnRef->__call(string, array)
mw-error.log:#6 /workspace/src/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\DBConnRef->upsert(string, array, array, array, string)
mw-error.log:#7 /workspace/src/extensions/CheckUser/src/Jobs/UpdateUserCentralIndexJob.php(60): Wikimedia\Rdbms\InsertQueryBuilder->execute()
mw-error.log:#8 /workspace/src/includes/jobqueue/JobRunner.php(369): MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob->run()
mw-error.log:#9 /workspace/src/includes/jobqueue/JobRunner.php(330): JobRunner->doExecuteJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-error.log:#10 /workspace/src/includes/jobqueue/JobRunner.php(225): JobRunner->executeJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-error.log:#11 /workspace/src/includes/specials/SpecialRunJobs.php(136): JobRunner->run(array)
mw-error.log:#12 /workspace/src/includes/specials/SpecialRunJobs.php(121): MediaWiki\Specials\SpecialRunJobs->doRun(array)
mw-error.log:#13 /workspace/src/includes/specialpage/SpecialPage.php(719): MediaWiki\Specials\SpecialRunJobs->execute(null)
mw-error.log:#14 /workspace/src/includes/specialpage/SpecialPageFactory.php(1708): MediaWiki\SpecialPage\SpecialPage->run(null)
mw-error.log:#15 /workspace/src/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
mw-error.log:#16 /workspace/src/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
mw-error.log:#17 /workspace/src/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
mw-error.log:#18 /workspace/src/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
mw-error.log:#19 {main}
mw-error.log:2024-09-06 13:59:10 a6cfaed359e0 wikidb: [bbc7172e4b3a78e2b00ac367] /index.php   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction
mw-error.log:Function: MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob::run
mw-error.log:Query: INSERT INTO `cuci_user` (ciu_timestamp,ciu_ciwm_id,ciu_central_id) VALUES ('20240906135836',1,1086) ON DUPLICATE KEY UPDATE ciu_timestamp = '20240906135836'
mw-error.log:#0 /workspace/src/includes/libs/rdbms/database/Database.php(1177): Wikimedia\Rdbms\Database->getQueryException(string, int, string, string)
mw-error.log:#1 /workspace/src/includes/libs/rdbms/database/Database.php(1151): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, int, string, string)
mw-error.log:#2 /workspace/src/includes/libs/rdbms/database/Database.php(643): Wikimedia\Rdbms\Database->reportQueryError(string, int, string, string, bool)
mw-error.log:#3 /workspace/src/includes/libs/rdbms/database/DatabaseMySQL.php(542): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
mw-error.log:#4 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\DatabaseMySQL->upsert(string, array, array, array, string)
mw-error.log:#5 /workspace/src/includes/libs/rdbms/database/DBConnRef.php(567): Wikimedia\Rdbms\DBConnRef->__call(string, array)
mw-error.log:#6 /workspace/src/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\DBConnRef->upsert(string, array, array, array, string)
mw-error.log:#7 /workspace/src/extensions/CheckUser/src/Jobs/UpdateUserCentralIndexJob.php(60): Wikimedia\Rdbms\InsertQueryBuilder->execute()
mw-error.log:#8 /workspace/src/includes/jobqueue/JobRunner.php(369): MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob->run()
mw-error.log:#9 /workspace/src/includes/jobqueue/JobRunner.php(330): JobRunner->doExecuteJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-error.log:#10 /workspace/src/includes/jobqueue/JobRunner.php(225): JobRunner->executeJob(MediaWiki\CheckUser\Jobs\UpdateUserCentralIndexJob)
mw-error.log:#11 /workspace/src/includes/specials/SpecialRunJobs.php(136): JobRunner->run(array)
mw-error.log:#12 /workspace/src/includes/specials/SpecialRunJobs.php(121): MediaWiki\Specials\SpecialRunJobs->doRun(array)
mw-error.log:#13 /workspace/src/includes/specialpage/SpecialPage.php(719): MediaWiki\Specials\SpecialRunJobs->execute(null)
mw-error.log:#14 /workspace/src/includes/specialpage/SpecialPageFactory.php(1708): MediaWiki\SpecialPage\SpecialPage->run(null)
mw-error.log:#15 /workspace/src/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
mw-error.log:#16 /workspace/src/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
mw-error.log:#17 /workspace/src/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
mw-error.log:#18 /workspace/src/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
mw-error.log:#19 {main}

Occasional CI failures due to deadlocks are nothing new, but as this is a relatively new table (T373018) and a risk of deadlocks was already mentioned (T368151#9921298), this still seems worth investigating.

Details

SubjectRepoBranchLines +/-
Get an scoped lock instead of using FOR UPDATE lockmediawiki/extensions/CheckUsermaster+8 -1
Customize query in gerrit

Related Objects

Event Timeline

Lucas_Werkmeister_WMDE created this task.Sep 6 2024, 2:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2024, 2:21 PM
Dreamy_Jazz edited projects, added CheckUser; removed CheckUser-GlobalContributions.Sep 6 2024, 2:28 PM
kostajh subscribed.Sep 6 2024, 2:31 PM
Lucas_Werkmeister_WMDE added a comment.Sep 6 2024, 2:53 PM

Hm, it failed again in this build for the same Gerrit change… is it a consistent failure or did we just get unlucky twice in a row?

Dreamy_Jazz added a comment.Sep 6 2024, 3:05 PM

Either way, we should probably fix this.

I have a suggested patch I'm about to upload that we could mark as depends on and see if it fixes the issue.

gerritbot added a comment.Sep 6 2024, 3:08 PM

Change #1071228 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/CheckUser@master] Get an scoped lock instead of using FOR UPDATE lock

https://gerrit.wikimedia.org/r/1071228

gerritbot added a project: Patch-For-Review.Sep 6 2024, 3:08 PM
Lucas_Werkmeister_WMDE added a comment.Sep 6 2024, 3:16 PM

Yeah, it failed a third time, albeit with only one error in each log file this time. Let’s try Depends-On then.

Lucas_Werkmeister_WMDE added a comment.EditedSep 6 2024, 3:18 PM

Oops, sorry, I mixed up Gerrit changes – that last build was actually on another change. I guess that confirms that it’s not specific to the one Wikibase change ;)

Anyway, I’ve added the Depends-On there, let’s see if it works? (The CheckUser change looks reasonable to me but I don’t think I should be the one to +2 it…)

Lucas_Werkmeister_WMDE added a comment.Sep 6 2024, 3:21 PM

Also a couple of failures on another Wikibase change that I didn’t see before but seem to be the same issue.

kostajh closed this task as Resolved.Sep 6 2024, 3:34 PM
kostajh claimed this task.

Have +2'ed the patch, so tentatively resolving this, but please re-open if seen again. Sorry for the trouble, and thanks for filing the task @Lucas_Werkmeister_WMDE!

Dreamy_Jazz claimed this task.Sep 6 2024, 3:34 PM
gerritbot added a comment.Sep 6 2024, 4:21 PM

Change #1071228 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Get an scoped lock instead of using FOR UPDATE lock

https://gerrit.wikimedia.org/r/1071228

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2024, 4:30 PM
ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.22; 2024-09-10).Sep 6 2024, 5:00 PM
Dreamy_Jazz changed the task status from Duplicate to Resolved.Sep 6 2024, 6:38 PM
Dreamy_Jazz closed this task as a duplicate of T374271: UpdateUserCentralIndexJob fails with deadlock on table cuci_user on CI run for apitests on Wikibase.
Dreamy_Jazz merged a task: T374271: UpdateUserCentralIndexJob fails with deadlock on table cuci_user on CI run for apitests on Wikibase.
Dreamy_Jazz added a subscriber: Umherirrender.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits