Page MenuHomePhabricator

Unblock stuck global renames at Meta-Wiki
Closed, ResolvedPublic

Description

https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress displays that there are 13 global renames stuck, all, at meta.wikimedia.

When the job tried to run on Meta it seems that it has failed there for some reason and didn't continued on the rest of the projects.

Requesting that the logs be checked to see if the jobs are still trying to restart themselves, if not, requeue the jobs and investigate why this has happened.

Link to guide: https://wikitech.wikimedia.org/wiki/Stuck_global_renames

Event Timeline

Restricted Application added a project: User-MarcoAurelio. · View Herald TranscriptAug 16 2017, 2:27 PM
Restricted Application added subscribers: Dereckson, Aklapper. · View Herald Transcript
Marostegui added a subscriber: Marostegui.

I am removing the DBA tag, as there is not much for us to do here :-)
Once you have the tasks for each rename, if they need to happen, ping us as you normally do!

Thanks

MarcoAurelio renamed this task from 13 global renames stuck to Unblock 13 global renames stuck at Meta-Wiki and elsewhere.Aug 16 2017, 2:34 PM
MarcoAurelio updated the task description. (Show Details)
RuyP added a subscriber: RuyP.Aug 16 2017, 2:58 PM

Just out of curiosity. Would it be possible to prevent this from occurring again by blocking further renames whenever there was already 10 renames in progress?

A new global rename has become stuck just now: https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Roquetero

All going good until it hit metawiki.

MarcoAurelio added a subscriber: Platonides.
This comment was removed by MarcoAurelio.
MarcoAurelio renamed this task from Unblock 13 global renames stuck at Meta-Wiki and elsewhere to Unblock stuck global renames at Meta-Wiki.Aug 16 2017, 7:39 PM

Just to clarify: it is not that those renames are stuck, which they are, but that every global rename is becoming stuck now and then at metawiki for some reason we don't know.

Hmm, this is weird. I don't see any relevant exceptions in exception.log for the past two days. And when I run:

legoktm@terbium:~$ mwscript showJobs.php --wiki=metawiki --group
translationNotificationJob: 0 queued; 949 claimed (0 active, 949 abandoned); 0 delayed
refreshLinks: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed
deleteLinks: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed
htmlCacheUpdate: 1 queued; 0 claimed (0 active, 0 abandoned); 0 delayed
sendMail: 0 queued; 42 claimed (0 active, 42 abandoned); 0 delayed
refreshLinksPrioritized: 2 queued; 0 claimed (0 active, 0 abandoned); 0 delayed
categoryMembershipChange: 0 queued; 32 claimed (0 active, 32 abandoned); 0 delayed
cdnPurge: 0 queued; 0 claimed (0 active, 0 abandoned); 115 delayed
cirrusSearchIncomingLinkCount: 3 queued; 0 claimed (0 active, 0 abandoned); 1 delayed
cirrusSearchCheckerJob: 1 queued; 0 claimed (0 active, 0 abandoned); 649 delayed
MessageUpdateJob: 0 queued; 1372 claimed (0 active, 1372 abandoned); 0 delayed
MessageGroupStatesUpdaterJob: 0 queued; 1 claimed (1 active, 0 abandoned); 0 delayed

There's nothing about renames in that, and if they had failed they should be either queued or abandoned...

Matiia added a subscriber: Matiia.Aug 17 2017, 1:47 AM

translationNotificationJob: 0 queued; 949 claimed (0 active, 949 abandoned); 0 delayed is probably due to T160276#3518398 but that was some days ago.

Something strange seems to be happening on Meta-Wiki: accounts are not being autocreated there according to @Vituzzu, all renames get stuck when they hit Meta-Wiki, etc.

Can you try to requeue one rename and see what happens?

Base added a subscriber: Base.Aug 17 2017, 2:30 PM
MarcoAurelio added a comment.EditedAug 17 2017, 8:54 PM

Please also note that one of the accounts whose rename is in progress has wrote to me on my talk page at eswiki. That is weird. Ain't accounts supposed to remain locked until the rename job finished? If so, this is not working either, and therefore there is risk of broken global accounts due to autocreations pending the finish of the rename.

It seems to me that the stuck renames ain't the problem but another symptom derived of whatever is broken and causing things to break around.

I think, therefore, that this should be a UBN Task.

Update: the signature on the talk page fooled me; it was an alternate username. The other parts of this statement not struck are still valid.

This task should be solved as fast as you can! many new requests accumulate!

@alanajjar: Please avoid adding "+1" / "me too" comments that do not bring a task closer to resolution and create notifications for everybody. Thanks a lot! :)

MarcoAurelio raised the priority of this task from High to Unbreak Now!.Aug 19 2017, 11:36 AM

Per my statement at T173419#3532167 and given that this is having production impact, and also that it was a feature that this was working fine previously and now it is not working at all I am raising the priority of this task to UBN. We've currently have 14 accounts locked due to unfinished renames and still don't know why all renames ain't even trying to run at Meta-Wiki. We should investigate what has been deployed on MW, extensions and mediawiki-config last week and seee what might be the cause for that.

I suggest that we try unblocking the oldest queued rename in Special:GlobalRenameProgress and see what the logs return in verbose mode. That might give us some hints IMHO.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptAug 19 2017, 11:36 AM
How-To
# Verify if there are running rename jobs
you@terbium:~$ mwscript showJobs.php --wiki=metawiki --group

# Unblock the rename of Bolsée to Kathmandu2017
you@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017"

# use the script above with --ignorestatus if it fails

Mentioned in SAL (#wikimedia-operations) [2017-08-19T12:47:15Z] <Amir1> ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)

The first one is gone now, but others are not being picked up it seems.

Mentioned in SAL (#wikimedia-operations) [2017-08-19T13:00:46Z] <Amir1> running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)

Mentioned in SAL (#wikimedia-operations) [2017-08-19T13:17:13Z] <Amir1> another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井 俊光 → Toshimit) (T173419)

MarcoAurelio lowered the priority of this task from Unbreak Now! to Normal.EditedAug 19 2017, 1:36 PM

Reduce to normal as the queue has been cleared of stuck renames, and it seems the jobs are being run this time. Leaving open for further investigation.

MarcoAurelio closed this task as Resolved.Aug 20 2017, 9:31 AM
MarcoAurelio assigned this task to Ladsgroup.

Closed as everything seems back to normal. Thank you for your help.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 20 2017, 9:31 AM
MarcoAurelio moved this task from radar to cabinet on the User-MarcoAurelio board.Aug 20 2017, 9:31 AM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM