Special:GlobalRenameProgress/MB is stuck
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	K6ka
	Jul 31 2016, 6:26 PM

Description

https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/MB has been stuck for nearly four days now. The rename on enwiki is stuck at "In progress" but doesn't seem to actually be progressing. All other wikis have been renamed successfully.

Details

	Subject	Repo	Branch	Lines +/-
	Increase retries for rename jobs	operations/puppet	production	+4 -1

Customize query in gerrit

Related Objects

Mentioned In: rOPUP10567d0c1a16: Increase retries for rename jobs
rOPUPc1c56a3a779b: Increase retries for rename jobs
rOPUP2610b1bc8c50: Increase retries for rename jobs

Event Timeline

K6ka created this task.Jul 31 2016, 6:26 PM

Restricted Application added subscribers: Steinsplitter, Aklapper. · View Herald TranscriptJul 31 2016, 6:26 PM

Peachey88 added a subscriber: Legoktm.Aug 1 2016, 5:28 AM

Peachey88 merged a task: T141819: Unblock global rename of Mb66w to MB.Aug 2 2016, 2:58 AM

Peachey88 added a project: Wikimedia-Site-requests.

Peachey88 added subscribers: MarcoAurelio, Tgr, Matanya and 2 others.

MarcoAurelio triaged this task as Unbreak Now! priority.Aug 2 2016, 7:06 AM

Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptAug 2 2016, 7:06 AM

MarcoAurelio awarded a token.Aug 2 2016, 3:23 PM

All three job retry attempts failed with a lock wait timeout (twice when changing usernames in page revisions, once on SELECT wl_user, wl_notificationtimestamp FROM watchlist WHERE wl_namespace = '2' AND wl_title = 'Mb66w' FOR UPDATE - how on Earth can you get lock contention on that?).

Trijnstel subscribed.Aug 2 2016, 10:12 PM

In T141731#2516945, @Tgr wrote:

All three job retry attempts failed with a lock wait timeout (twice when changing usernames in page revisions, once on SELECT wl_user, wl_notificationtimestamp FROM watchlist WHERE wl_namespace = '2' AND wl_title = 'Mb66w' FOR UPDATE - how on Earth can you get lock contention on that?).

No idea. However it'd be good if in these cases the GlobalRenameProgress table showed a "failed" status instead of "in progress", which ain't true. Thanks.

Failed jobs are retried by the job queue; I'm not sure if there is a way for a job to tell that it has failed for the last time. Maybe renameuser_status could get a failure count column?

@aaron, any thoughts? Global rename cannot be made atomic (as it consists of a bunch of jobs running on a bunch of different wikis), can we increase the lock timeout somehow?

Maybe we can change the puppet config to bump the retry count for rename related jobs? E.g. in modules/mediawiki/templates/jobrunner/jobrunner.conf.erb .

Change 302650 had a related patch set uploaded (by Gergő Tisza):
Increase retries for rename jobs

https://gerrit.wikimedia.org/r/302650

Tgr mentioned this in rOPUP2610b1bc8c50: Increase retries for rename jobs.Aug 3 2016, 2:39 AM

This task is set as UBN! Is that correct @Tgr / @aaron ? If so, will the above patch from @Tgr fix the issue and what is the timeline for deploying?

Nope, a single case of failed rename in two weeks is not an emergency. This is a puppet patch so I can get it deployed in tomorrow's SWAT (or someone can just go on and merge it I guess). I'm working on fixing the affected account.

Thanks!

Tgr mentioned this in rOPUPc1c56a3a779b: Increase retries for rename jobs.Aug 4 2016, 2:16 AM

The global rename seemed to be finished so I reattached the enwiki account and deleted the status row. MB should be OK now.

User reported that they were able to log in. https://en.wikipedia.org/w/index.php?title=User_talk:K6ka&diff=733020609&oldid=732921752

I'm closing this since the issue with this rename is done and the task is specific to this. Should you disagree, feel free to reopen :)

MarcoAurelio moved this task from Backlog to Closed on the GlobalRename board.Aug 7 2016, 9:09 PM

MarcoAurelio moved this task from Backlog to Done on the MediaWiki-extensions-CentralAuth board.

MarcoAurelio moved this task from Backlog to Done on the Wikimedia-Site-requests board.

Change 302650 merged by Filippo Giunchedi:
Increase retries for rename jobs