Page MenuHomePhabricator
Create Task
Maniphest T173462

LinksUpdate::acquirePageLock: Cannot flush pre-lock snapshot because writes are pending
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

This started with the rollout of 1.30.0-wmf.14 to group1 wikis

#0 /srv/mediawiki/php-1.30.0-wmf.14/includes/deferred/LinksUpdate.php(203): Wikimedia\Rdbms\Database->getScopedLockAndFlush(string, string, integer)
#1 /srv/mediawiki/php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php(144): LinksUpdate::acquirePageLock(Wikimedia\Rdbms\DatabaseMysqli, integer, string)
#2 /srv/mediawiki/php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php(118): RefreshLinksJob->runForTitle(Title)
#3 /srv/mediawiki/php-1.30.0-wmf.14/includes/jobqueue/JobRunner.php(295): RefreshLinksJob->run()
#4 /srv/mediawiki/php-1.30.0-wmf.14/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(RefreshLinksJob, Wikimedia\Rdbms\LBFactoryMulti, BufferingStatsdDataFactory, integer)
#5 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
#6 {main}

Details

SubjectRepoBranchLines +/-
Avoid lock acquisition errors for multi-title refreshlinks jobsmediawiki/coremaster+4 -0
Avoid lock acquisition errors for multi-title refreshlinks jobsmediawiki/corewmf/1.30.0-wmf.14+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

thcipriani created this task.Aug 16 2017, 9:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 16 2017, 9:10 PM
thcipriani added a parent task: T170632: 1.30.0-wmf.14 deployment blockers.Aug 16 2017, 9:16 PM
Stashbot subscribed.Aug 16 2017, 9:21 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-16T21:21:28Z] <thcipriani@tin> Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)

Stashbot added a comment.Aug 16 2017, 9:22 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-16T21:22:09Z] <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462

thcipriani triaged this task as Unbreak Now! priority.Aug 16 2017, 9:25 PM

UBN! since I added as a blocker to wmf.14.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptAug 16 2017, 9:25 PM
Stashbot added a comment.Aug 16 2017, 9:37 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-16T21:37:28Z] <thcipriani> train is on hold pending resolution of T173462

greg added a project: MediaWiki-Core-JobQueue.Aug 16 2017, 10:03 PM
greg added subscribers: aaron, greg.

JobRunner/Queue -> @aaron

thcipriani added a comment.Aug 16 2017, 10:17 PM

So while this error started at a low-level when I rolled forward group1 wikis, it is now present on wmf.13 since rollback: https://logstash.wikimedia.org/goto/cbfaa6baf81c7edecfd8e9a2968966d3

Jrbranaa added a subtask: T173477: wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending.Aug 16 2017, 11:54 PM
Krinkle added a project: Performance-Team.Aug 17 2017, 1:08 AM
Krinkle moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.
Ladsgroup subscribed.Aug 17 2017, 7:04 AM

Might be related to T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis and its subtasks. Can't say for sure.

gerritbot subscribed.Aug 17 2017, 9:02 AM

Change 372345 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] [WIP] Avoid lock acquisition errors for multi-title refreshlinks jobs

https://gerrit.wikimedia.org/r/372345

gerritbot added a project: Patch-For-Review.Aug 17 2017, 9:02 AM
gerritbot added a comment.Aug 17 2017, 4:39 PM

Change 372414 had a related patch set uploaded (by Thcipriani; owner: Aaron Schulz):
[mediawiki/core@wmf/1.30.0-wmf.14] Avoid lock acquisition errors for multi-title refreshlinks jobs

https://gerrit.wikimedia.org/r/372414

gerritbot added a comment.Aug 17 2017, 4:49 PM

Change 372414 merged by jenkins-bot:
[mediawiki/core@wmf/1.30.0-wmf.14] Avoid lock acquisition errors for multi-title refreshlinks jobs

https://gerrit.wikimedia.org/r/372414

Stashbot added a comment.Aug 17 2017, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-17T16:51:30Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: [[gerrit:372414|Avoid lock acquisition errors for multi-title refreshlinks jobs]] T173462 (duration: 00m 51s)

gerritbot added a comment.Aug 17 2017, 4:55 PM

Change 372345 merged by jenkins-bot:
[mediawiki/core@master] Avoid lock acquisition errors for multi-title refreshlinks jobs

https://gerrit.wikimedia.org/r/372345

greg added a comment.Aug 17 2017, 5:32 PM

We now had to rollback to wmf.13 again due to logged in users getting 503s due to T173520.

thcipriani closed this task as Resolved.Aug 17 2017, 7:34 PM
thcipriani assigned this task to aaron.
In T173462#3531364, @greg wrote:

We now had to rollback to wmf.13 again due to logged in users getting 503s due to T173520.

Rolled forward again (with @aaron's patch in place) and this error no longer appears. Closing resolved.

Liuxinyu970226 unsubscribed.Aug 18 2017, 1:04 PM
ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-08-15 (1.30.0-wmf.14)).Aug 21 2017, 7:01 PM
Jrbranaa closed subtask T173477: wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending as Resolved.Sep 12 2017, 10:08 PM
Krinkle added a parent task: T175897: Audit and improve JobQueue stability and performance (2017).Sep 14 2017, 8:54 AM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits