https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress displays that there are 13 global renames stuck, all, at meta.wikimedia.
When the job tried to run on Meta it seems that it has failed there for some reason and didn't continued on the rest of the projects.
Requesting that the logs be checked to see if the jobs are still trying to restart themselves, if not, requeue the jobs and investigate why this has happened.
Link to guide: https://wikitech.wikimedia.org/wiki/Stuck_global_renames
I am removing the DBA tag, as there is not much for us to do here :-)
Once you have the tasks for each rename, if they need to happen, ping us as you normally do!
Thanks
Just out of curiosity. Would it be possible to prevent this from occurring again by blocking further renames whenever there was already 10 renames in progress?
A new global rename has become stuck just now: https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Roquetero
All going good until it hit metawiki.
Just to clarify: it is not that those renames are stuck, which they are, but that every global rename is becoming stuck now and then at metawiki for some reason we don't know.
Hmm, this is weird. I don't see any relevant exceptions in exception.log for the past two days. And when I run:
legoktm@terbium:~$ mwscript showJobs.php --wiki=metawiki --group translationNotificationJob: 0 queued; 949 claimed (0 active, 949 abandoned); 0 delayed refreshLinks: 0 queued; 2 claimed (1 active, 1 abandoned); 0 delayed deleteLinks: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed htmlCacheUpdate: 1 queued; 0 claimed (0 active, 0 abandoned); 0 delayed sendMail: 0 queued; 42 claimed (0 active, 42 abandoned); 0 delayed refreshLinksPrioritized: 2 queued; 0 claimed (0 active, 0 abandoned); 0 delayed categoryMembershipChange: 0 queued; 32 claimed (0 active, 32 abandoned); 0 delayed cdnPurge: 0 queued; 0 claimed (0 active, 0 abandoned); 115 delayed cirrusSearchIncomingLinkCount: 3 queued; 0 claimed (0 active, 0 abandoned); 1 delayed cirrusSearchCheckerJob: 1 queued; 0 claimed (0 active, 0 abandoned); 649 delayed MessageUpdateJob: 0 queued; 1372 claimed (0 active, 1372 abandoned); 0 delayed MessageGroupStatesUpdaterJob: 0 queued; 1 claimed (1 active, 0 abandoned); 0 delayed
There's nothing about renames in that, and if they had failed they should be either queued or abandoned...
translationNotificationJob: 0 queued; 949 claimed (0 active, 949 abandoned); 0 delayed is probably due to T160276#3518398 but that was some days ago.
Something strange seems to be happening on Meta-Wiki: accounts are not being autocreated there according to @Vituzzu, all renames get stuck when they hit Meta-Wiki, etc.
Can you try to requeue one rename and see what happens?
Please also note that one of the accounts whose rename is in progress has wrote to me on my talk page at eswiki. That is weird. Ain't accounts supposed to remain locked until the rename job finished? If so, this is not working either, and therefore there is risk of broken global accounts due to autocreations pending the finish of the rename.
It seems to me that the stuck renames ain't the problem but another symptom derived of whatever is broken and causing things to break around.
I think, therefore, that this should be a UBN Task.
Update: the signature on the talk page fooled me; it was an alternate username. The other parts of this statement not struck are still valid.
@alanajjar: Please avoid adding "+1" / "me too" comments that do not bring a task closer to resolution and create notifications for everybody. Thanks a lot! :)
Per my statement at T173419#3532167 and given that this is having production impact, and also that it was a feature that this was working fine previously and now it is not working at all I am raising the priority of this task to UBN. We've currently have 14 accounts locked due to unfinished renames and still don't know why all renames ain't even trying to run at Meta-Wiki. We should investigate what has been deployed on MW, extensions and mediawiki-config last week and seee what might be the cause for that.
I suggest that we try unblocking the oldest queued rename in Special:GlobalRenameProgress and see what the logs return in verbose mode. That might give us some hints IMHO.
# Verify if there are running rename jobs you@terbium:~$ mwscript showJobs.php --wiki=metawiki --group # Unblock the rename of Bolsée to Kathmandu2017 you@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" # use the script above with --ignorestatus if it fails
Mentioned in SAL (#wikimedia-operations) [2017-08-19T12:47:15Z] <Amir1> ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
Mentioned in SAL (#wikimedia-operations) [2017-08-19T13:00:46Z] <Amir1> running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
Mentioned in SAL (#wikimedia-operations) [2017-08-19T13:17:13Z] <Amir1> another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井 俊光 → Toshimit) (T173419)
Reduce to normal as the queue has been cleared of stuck renames, and it seems the jobs are being run this time. Leaving open for further investigation.