Page MenuHomePhabricator

Updates for "wl_notificationtimestamp" after page views fail sometimes due to "Lock wait timeout exceeded"
Open, Needs TriagePublic

Description

Error

Request ID: W5@o9grAAEAAADq-rAwAAAAK

message
Query: UPDATE  `watchlist` SET wl_notificationtimestamp = '##############' WHERE wl_user IN ('###, '###', ..., '###')  AND wl_namespace = '0' AND wl_title = '***'
Function: WatchedItemStore::updateNotificationTimestamp
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.192.48.114)
stacktrace
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/Database.php(2074): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/DBConnRef.php(49): Wikimedia\Rdbms\Database->update(string, array, array, string)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/rdbms/database/DBConnRef.php(309): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/watcheditem/WatchedItemStore.php(833): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/MWCallableUpdate.php(34): Closure$WatchedItemStore::updateNotificationTimestamp()
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(268): MWCallableUpdate->doUpdate()
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(226): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(134): DeferredUpdates::execute(array, string, integer)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(914): DeferredUpdates::doUpdates(string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(734): MediaWiki->restInPeace(string, boolean)

Notes

Happens on may different wikis, although majority from commons.wikimedia.org and vi.wikipedia.org. It happens on page views, EditPage edits, and API edits.

The query looks similar to T204555, but comes from different code triggered by different circumstances. It might be that both can be fixed in the same way, but they will probably each require a separate fix.

See also:

Event Timeline

Krinkle created this task.Sep 17 2018, 4:49 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 17 2018, 4:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle renamed this task from WatchedItemStore::updateNotificationTimestamp fails with "Lock wait timeout exceeded" on watchlist.wl_notificationtimestamp to Updates for "wl_notificationtimestamp" after page views fail sometimes due to "Lock wait timeout exceeded".Oct 2 2018, 9:12 PM

Still seen on 1.32.0-wmf.23. The error message and code path changed slightly, so here's an updated trace for search results:

Request ID: W7PKcQrAIFYAAHtya68AAAAP

A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: UPDATE  `watchlist` SET wl_notificationtimestamp = '20181002194345' WHERE wl_user = '#####' AND wl_namespace = '0' AND wl_title = '#####' AND (wl_notificationtimestamp < '20181002194345')
Function: ActivityUpdateJob::updateWatchlistNotification
Error: 1205 Lock wait timeout exceeded; try restarting transaction

#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/database/Database.php(1198): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.32.0-wmf.23/includes/libs/rdbms/database/Database.php(2074): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/jobqueue/jobs/ActivityUpdateJob.php(80): Wikimedia\Rdbms\Database->update(string, array, array, string)
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/jobqueue/jobs/ActivityUpdateJob.php(49): ActivityUpdateJob->updateWatchlistNotification()
#5 /srv/mediawiki/php-1.32.0-wmf.23/includes/watcheditem/WatchedItemStore.php(893): ActivityUpdateJob->run()
#6 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/MWCallableUpdate.php(34): Closure$WatchedItemStore::resetNotificationTimestamp()
#7 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(268): MWCallableUpdate->doUpdate()
#8 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#9 /srv/mediawiki/php-1.32.0-wmf.23/includes/deferred/DeferredUpdates.php(134): DeferredUpdates::execute(array, string, integer)
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(914): DeferredUpdates::doUpdates(string)
#11 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(734): MediaWiki->restInPeace(string, boolean)

@Catrope will take a look at this

Never mind, I got confused between this task and T204555

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM

Like T208542, I see 4 instances of this in the last 60 days so I'm going to mark this as Triaged but Future on our board as I don't think Growth-Team would be able to prioritize it in the short-to-medium term.

kostajh moved this task from Inbox to Triaged but Future on the Growth-Team board.Aug 29 2019, 7:43 AM