Deadlocks from job setting VectorSkinVersion user preference to 1
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

mwversion: 1.38.0-wmf.7
reqId: d38012fb-8ccf-41d3-a63b-4555e75055e5
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1136)
Function: MediaWiki\User\UserOptionsManager::saveOptionsInternal
Query: INSERT IGNORE INTO `user_properties`

exception.trace

from /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(1830)
#0 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(1814): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(1789): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(1327): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(2579): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/Database.php(2539): Wikimedia\Rdbms\Database->doInsertNonConflicting(string, array, string)
#5 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->insert(string, array, string, array)
#6 /srv/mediawiki/php-1.38.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(380): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.38.0-wmf.7/includes/user/UserOptionsManager.php(465): Wikimedia\Rdbms\DBConnRef->insert(string, array, string, array)
#8 /srv/mediawiki/php-1.38.0-wmf.7/includes/user/User.php(3286): MediaWiki\User\UserOptionsManager->saveOptionsInternal(User, Wikimedia\Rdbms\MaintainableDBConnRef)
#9 /srv/mediawiki/php-1.38.0-wmf.7/extensions/Wikibase/client/includes/Hooks/EchoNotificationsHandlers.php(137): User->saveSettings()
#10 /srv/mediawiki/php-1.38.0-wmf.7/extensions/Wikibase/client/includes/Hooks/EchoNotificationsHandlers.php(123): Wikibase\Client\Hooks\EchoNotificationsHandlers->doLocalUserCreated(User, boolean)
#11 /srv/mediawiki/php-1.38.0-wmf.7/includes/HookContainer/HookContainer.php(338): Wikibase\Client\Hooks\EchoNotificationsHandlers::onLocalUserCreated(User, boolean)
#12 /srv/mediawiki/php-1.38.0-wmf.7/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#13 /srv/mediawiki/php-1.38.0-wmf.7/includes/HookContainer/HookRunner.php(2366): MediaWiki\HookContainer\HookContainer->run(string, array)
#14 /srv/mediawiki/php-1.38.0-wmf.7/includes/auth/AuthManager.php(1828): MediaWiki\HookContainer\HookRunner->onLocalUserCreated(User, boolean)
#15 /srv/mediawiki/php-1.38.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUtilityService.php(126): MediaWiki\Auth\AuthManager->autoCreateUser(User, string, boolean, boolean)
#16 /srv/mediawiki/php-1.38.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUtils.php(105): CentralAuthUtilityService->autoCreateUser(User)
#17 /srv/mediawiki/php-1.38.0-wmf.7/extensions/CentralAuth/includes/CentralAuthCreateLocalAccountJob.php(81): CentralAuthUtils::autoCreateUser(User)
#18 /srv/mediawiki/php-1.38.0-wmf.7/extensions/EventBus/includes/JobExecutor.php(79): CentralAuthCreateLocalAccountJob->run()
#19 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#20 {main}

Impact

Presumably, some requests that involve changes to user preferences error out.

Notes

Logs: https://logstash.wikimedia.org/goto/b7394e8c496b08aa6629cb1f5835a6e5
Started at 19:50 UTC so probably train-related.

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

	Subject	Repo	Branch	Lines +/-
	End migration mode	mediawiki/skins/Vector	master	+112 -1 K

Customize query in gerrit

Related Objects

Mentioned In: T306636: DBQueryError: Error 1213: Deadlock found in UserOptionsManager::saveOptionsInternal

Event Timeline

Tgr created this task.Nov 4 2021, 12:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2021, 12:54 AM

Zabe subscribed.Nov 4 2021, 8:48 AM

Jdlrobson added a project: Web-Team-Backlog (Needs Prioritization (Tech)).Nov 9 2021, 2:09 AM

Jdlrobson moved this task from Backlog to [active] Split Vector skins on the Web-Team-Backlog (Needs Prioritization (Tech)) board.Nov 10 2021, 10:21 PM

thcipriani moved this task from Untriaged to Nov 2021 on the Wikimedia-production-error board.Nov 18 2021, 4:53 PM

I only see one instance of this error in the last 4 months.
https://logstash.wikimedia.org/goto/7e46895d90b46cf93349fa33b4097153

Not sure what web team can do about this one as it seems to relate to preferences rather than Vector code.

Jdlrobson moved this task from Tracking Indefinitely to [active] Split Vector skins on the Web-Team-Backlog (Needs Prioritization (Tech)) board.Jan 7 2022, 4:46 PM

Jdlrobson moved this task from Needs triage to Technical on the Vector (legacy skin) board.Jan 27 2022, 4:21 PM

This should be addressed now as the skin version preference is no longer actively set. Also no errors in last 3 months. Please reopen if you see this again.

Showed up today.

[2898c529-42d5-4826-83bd-2924e7f7cafd] /wiki/Special:CreateLocalAccount   Wikimedia\Rdbms\DBQueryError: Error 1213: Deadlock found when trying to get lock; try restarting transaction (db1118)
Function: MediaWiki\User\UserOptionsManager::saveOptionsInternal
Query: INSERT IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (43539902,'VectorSkinVersion','1')

from /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(1606)
#0 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(1590): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(1564): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(1173): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(2370): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/Database.php(2330): Wikimedia\Rdbms\Database->doInsertNonConflicting(string, array, string)
#5 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/DBConnRef.php(69): Wikimedia\Rdbms\Database->insert(string, array, string, array)
#6 /srv/mediawiki/php-1.38.0-wmf.25/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.38.0-wmf.25/includes/user/UserOptionsManager.php(465): Wikimedia\Rdbms\DBConnRef->insert(string, array, string, array)
#8 /srv/mediawiki/php-1.38.0-wmf.25/includes/user/User.php(2830): MediaWiki\User\UserOptionsManager->saveOptionsInternal(User, Wikimedia\Rdbms\MaintainableDBConnRef)
#9 /srv/mediawiki/php-1.38.0-wmf.25/extensions/Wikibase/client/includes/Hooks/EchoNotificationsHandlers.php(144): User->saveSettings()
#10 /srv/mediawiki/php-1.38.0-wmf.25/extensions/Wikibase/client/includes/Hooks/EchoNotificationsHandlers.php(130): Wikibase\Client\Hooks\EchoNotificationsHandlers->doLocalUserCreated(User, boolean)
#11 /srv/mediawiki/php-1.38.0-wmf.25/includes/HookContainer/HookContainer.php(338): Wikibase\Client\Hooks\EchoNotificationsHandlers::onLocalUserCreated(User, boolean)
#12 /srv/mediawiki/php-1.38.0-wmf.25/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#13 /srv/mediawiki/php-1.38.0-wmf.25/includes/HookContainer/HookRunner.php(2370): MediaWiki\HookContainer\HookContainer->run(string, array)
#14 /srv/mediawiki/php-1.38.0-wmf.25/includes/auth/AuthManager.php(1828): MediaWiki\HookContainer\HookRunner->onLocalUserCreated(User, boolean)
#15 /srv/mediawiki/php-1.38.0-wmf.25/extensions/CentralAuth/includes/CentralAuthUtilityService.php(164): MediaWiki\Auth\AuthManager->autoCreateUser(User, string, boolean, boolean)
#16 /srv/mediawiki/php-1.38.0-wmf.25/extensions/CentralAuth/includes/User/CentralAuthForcedLocalCreationService.php(93): MediaWiki\Extension\CentralAuth\CentralAuthUtilityService->autoCreateUser(User, boolean)
#17 /srv/mediawiki/php-1.38.0-wmf.25/extensions/CentralAuth/includes/Special/SpecialCreateLocalAccount.php(88): MediaWiki\Extension\CentralAuth\User\CentralAuthForcedLocalCreationService->attemptAutoCreateLocalUserFromName(string, User, string)
#18 /srv/mediawiki/php-1.38.0-wmf.25/includes/htmlform/HTMLForm.php(726): MediaWiki\Extension\CentralAuth\Special\SpecialCreateLocalAccount->onSubmit(array, OOUIHTMLForm)
#19 /srv/mediawiki/php-1.38.0-wmf.25/includes/htmlform/HTMLForm.php(616): HTMLForm->trySubmit()
#20 /srv/mediawiki/php-1.38.0-wmf.25/includes/htmlform/HTMLForm.php(632): HTMLForm->tryAuthorizedSubmit()
#21 /srv/mediawiki/php-1.38.0-wmf.25/includes/specialpage/FormSpecialPage.php(209): HTMLForm->show()
#22 /srv/mediawiki/php-1.38.0-wmf.25/extensions/CentralAuth/includes/Special/SpecialCreateLocalAccount.php(59): FormSpecialPage->execute(NULL)
#23 /srv/mediawiki/php-1.38.0-wmf.25/includes/specialpage/SpecialPage.php(671): MediaWiki\Extension\CentralAuth\Special\SpecialCreateLocalAccount->execute(NULL)
#24 /srv/mediawiki/php-1.38.0-wmf.25/includes/specialpage/SpecialPageFactory.php(1378): SpecialPage->run(NULL)
#25 /srv/mediawiki/php-1.38.0-wmf.25/includes/MediaWiki.php(315): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#26 /srv/mediawiki/php-1.38.0-wmf.25/includes/MediaWiki.php(910): MediaWiki->performRequest()
#27 /srv/mediawiki/php-1.38.0-wmf.25/includes/MediaWiki.php(564): MediaWiki->main()
#28 /srv/mediawiki/php-1.38.0-wmf.25/index.php(53): MediaWiki->run()
#29 /srv/mediawiki/php-1.38.0-wmf.25/index.php(46): wfIndexMain()
#30 /srv/mediawiki/w/index.php(3): require(string)
#31 {main}

GeneralNotability subscribed.Mar 13 2022, 5:21 PM

In T294995#7771994, @Zabe wrote:

Showed up today.

Noting that this does not happen all the time: the above stack trace was from a Special:CreateLocalAccount creation and I managed to use that same special page to create a new account on the same wiki (https://en.wikipedia.org/w/index.php?title=Special:Log&logid=128863499) without seeing that error message.

Confirming that it's inconsistent - I've done CreateLocalAccount several times today and it's only triggered once.

Change 755813 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] End migration mode

https://gerrit.wikimedia.org/r/755813

gerritbot added a project: Patch-For-Review.Mar 15 2022, 5:11 PM

We were still setting the skin preference inside onGetPreferences for backwards compatiblity. This has just been removed by https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/755813 after which this issue should be impossible.

Jdlrobson moved this task from Needs Analysis to Ready for Signoff on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.Mar 22 2022, 11:28 PM

In T294995#7798534, @Jdlrobson wrote:

We were still setting the skin preference inside onGetPreferences for backwards compatiblity. This has just been removed by https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/755813 after which this issue should be impossible.

I can still reproduce this issue when visiting guw.wikipedia.org from my staff account, at least for now. 0d2f325a-dfca-46f8-ae9f-036af9c33950 is an example reqId.

@Urbanecm that patch apparently didn't merge so isn't deployed.

In T294995#7800322, @Jdlrobson wrote:

@Urbanecm that patch apparently didn't merge so isn't deployed.

Oh, I didn't catch that little detail. In that case, looking forward to it getting merged :).

Change 755813 merged by jenkins-bot:

[mediawiki/skins/Vector@master] End migration mode

https://gerrit.wikimedia.org/r/755813

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2022, 5:10 PM

This error should now be impossible.

Change 774839 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@wmf/1.39.0-wmf.4] End migration mode

https://gerrit.wikimedia.org/r/774839

gerritbot added a project: Patch-For-Review.Mar 29 2022, 4:59 PM

brennen mentioned this in T306636: DBQueryError: Error 1213: Deadlock found in UserOptionsManager::saveOptionsInternal.Apr 21 2022, 3:46 PM

Deadlocks from job setting VectorSkinVersion user preference to 1Closed, ResolvedPublicPRODUCTION ERRORActions