Page MenuHomePhabricator
Create Task
Maniphest T204561

Updates for "wl_notificationtimestamp" after page views fail sometimes due to "Lock wait timeout exceeded"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

Request ID: W5@o9grAAEAAADq-rAwAAAAK

message
Query: UPDATE  `watchlist` SET wl_notificationtimestamp = '##############' WHERE wl_user IN ('###, '###', ..., '###')  AND wl_namespace = '0' AND wl_title = '***'
Function: WatchedItemStore::updateNotificationTimestamp
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.192.48.114)
stacktrace
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/Database.php(2074): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/DBConnRef.php(49): Wikimedia\Rdbms\Database->update(string, array, array, string)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/DBConnRef.php(309): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/watcheditem/WatchedItemStore.php(833): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/MWCallableUpdate.php(34): Closure$WatchedItemStore::updateNotificationTimestamp()
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(268): MWCallableUpdate->doUpdate()
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(226): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(134): DeferredUpdates::execute(array, string, integer)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(914): DeferredUpdates::doUpdates(string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(734): MediaWiki->restInPeace(string, boolean)

Notes

Happens on may different wikis, although majority from commons.wikimedia.org and vi.wikipedia.org. It happens on page views, EditPage edits, and API edits.

The query looks similar to T204555, but comes from different code triggered by different circumstances. It might be that both can be fixed in the same way, but they will probably each require a separate fix.

See also:

Details

SubjectRepoBranchLines +/-
ActivityUpdateJob: Use primary key for db update on watchlist tablemediawiki/coremaster+28 -11
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Sep 17 2018, 4:49 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 17 2018, 4:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle moved this task from Untriaged to Apr 2019 / 1.33.wmf.25+ on the Wikimedia-production-error board.Sep 17 2018, 4:49 PM
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle renamed this task from WatchedItemStore::updateNotificationTimestamp fails with "Lock wait timeout exceeded" on watchlist.wl_notificationtimestamp to Updates for "wl_notificationtimestamp" after page views fail sometimes due to "Lock wait timeout exceeded".Oct 2 2018, 9:12 PM
Krinkle added a subtask: T188801: Migrate wl_notificationtimestamp updates to the job queue.

Still seen on 1.32.0-wmf.23. The error message and code path changed slightly, so here's an updated trace for search results:

Request ID: W7PKcQrAIFYAAHtya68AAAAP

A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: UPDATE  `watchlist` SET wl_notificationtimestamp = '20181002194345' WHERE wl_user = '#####' AND wl_namespace = '0' AND wl_title = '#####' AND (wl_notificationtimestamp < '20181002194345')
Function: ActivityUpdateJob::updateWatchlistNotification
Error: 1205 Lock wait timeout exceeded; try restarting transaction

#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/database/Database.php(1198): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/database/Database.php(2074): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/jobqueue/jobs/ActivityUpdateJob.php(80): Wikimedia\Rdbms\Database->update(string, array, array, string)
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/jobqueue/jobs/ActivityUpdateJob.php(49): ActivityUpdateJob->updateWatchlistNotification()
#5 /srv/mediawiki/php-1.32.0-wmf.23/includes/watcheditem/WatchedItemStore.php(893): ActivityUpdateJob->run()
#6 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/MWCallableUpdate.php(34): Closure$WatchedItemStore::resetNotificationTimestamp()
#7 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(268): MWCallableUpdate->doUpdate()
#8 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#9 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(134): DeferredUpdates::execute(array, string, integer)
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(914): DeferredUpdates::doUpdates(string)
#11 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(734): MediaWiki->restInPeace(string, boolean)
Krinkle moved this task from Apr 2019 / 1.33.wmf.25+ to Older on the Wikimedia-production-error board.Oct 2 2018, 9:14 PM
Krinkle mentioned this in T208542: Watchlist reset fails "Lock wait timeout exceeded: WatchedItemStore::setNotificationTimestampsForUser" (from API setnotificationtimestamp).Nov 1 2018, 8:19 PM
aaron closed subtask T188801: Migrate wl_notificationtimestamp updates to the job queue as Resolved.Mar 15 2019, 5:11 PM
JTannerWMF added subscribers: Catrope, JTannerWMF.Mar 19 2019, 5:51 PM

@Catrope will take a look at this

Catrope added a comment.Mar 19 2019, 5:54 PM

Never mind, I got confused between this task and T204555

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
kostajh subscribed.Aug 29 2019, 7:42 AM

Like T208542, I see 4 instances of this in the last 60 days so I'm going to mark this as Triaged but Future on our board as I don't think Growth-Team would be able to prioritize it in the short-to-medium term.

kostajh moved this task from Inbox to Triaged but Future on the Growth-Team board.Aug 29 2019, 7:43 AM
MBinder_WMF added a project: Growth-Team-Filtering.Apr 15 2021, 6:56 PM
gerritbot added a comment.Aug 19 2021, 9:51 PM

Change 713944 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Use primary key for update on watchlist in ActivityUpdateJob

https://gerrit.wikimedia.org/r/713944

gerritbot added a project: Patch-For-Review.Aug 19 2021, 9:51 PM
gerritbot added a comment.Jan 27 2022, 4:21 AM

Change 713944 merged by jenkins-bot:

[mediawiki/core@master] ActivityUpdateJob: Use primary key for db update on watchlist table

https://gerrit.wikimedia.org/r/713944

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.20; 2022-01-31).Jan 27 2022, 5:00 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 27 2022, 5:10 AM
Umherirrender closed this task as Resolved.Jan 27 2022, 8:34 PM
Umherirrender claimed this task.
Umherirrender triaged this task as Lowest priority.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL