Page MenuHomePhabricator

Global rename of CodeCat → Rua: supervision needed
Closed, ResolvedPublic

Description

The main account has > 50k edits; (290000). See the request here. The request is accepted so we would appreciate if you could reserve for us a window so this rename can be performed and simultaneously supervised. Thanks.

Note: the user main edits number in en.wiktionary.org (280000 edits). (here)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2017, 3:50 PM
Marostegui triaged this task as Normal priority.Sep 28 2017, 3:51 PM
Marostegui moved this task from Triage to Backlog on the DBA board.
Marostegui added a subscriber: Marostegui.

When would you like to do this rename? In which timezone are you?

alanajjar raised the priority of this task from Normal to Needs Triage.Sep 28 2017, 3:52 PM
alanajjar added subscribers: MarcoAurelio, Legoktm, jcrespo, hoo.

@Marostegui if you available, we can do it now?

@Marostegui if you available, we can do it now?

Sure, give me 5 minutes to get ready

@Marostegui okay, when you being ready ping me here :)

@Marostegui if you available, we can do it now?

Sure, give me 5 minutes to get ready

Go for it!
Can you send me the meta progress URL so I can monitor it?

Mentioned in SAL (#wikimedia-operations) [2017-09-28T16:01:47Z] <marostegui> Global rename of CodeCat → Rua - T176985

@alanajjar any way to check the progress of the "in progress" status? :-)
It's been there for a while now

@Marostegui if you see the Central Auth of the new name (Rua) you'll found that it stooped in (en.wiktionary.org) then no progress after it!

@Marostegui if you see the Central Auth of the new name (Rua) you'll found that it stooped in (en.wiktionary.org) then no progress after it!

Yeah, that is what I was referring too, if there is any other way to see if it is actually doing something (as there is a large number of edits, it might be quite slow) or if it is stuck forever

@Marostegui I think there's a problem here!

If you see my contributions in en.wiktionary you'll notice there's no new contributions! so the process stop in en.wiktionary!!

Can you do anything? like start the process again?

@Marostegui I think there's a problem here!
If you see my contributions in en.wiktionary you'll notice there's no new contributions! so the process stop in en.wiktionary!!
Can you do anything? like start the process again?

Unfortunately there is nothing I can do as I am not familiar with this process. I just monitor the DB slaves to make sure there is no lag generated (and take measures to prevent it), but I don't really know how the process works underneath. You might need to ask around.
Maybe @Reedy happens to know something?

@MarcoAurelio could this be something related to T173419?
This rename got stuck in enwiktionary (where most of the edits are)

@MarcoAurelio could this be something related to T173419?
This rename got stuck in enwiktionary (where most of the edits are)

@MarcoAurelio what is your opinion here?

MarcoAurelio added a comment.EditedSep 28 2017, 5:13 PM

@Marostegui Please run

mwscript showJobs.php --wiki=enwiktionary --group

on terbium so we can know if there are queued jobs there.

Please also check LogStash or exception.log/fatalmonitor and tell us if there's any exception regarding rename.

@Marostegui Please run

mwscript showJobs.php --wiki=enwiktionary

on terbium so we can know if there are queued jobs there.

root@terbium:~# mwscript showJobs.php --wiki=enwiktionary
10602

I saw that this query timed out on enwiktionary - on the logs:

SELECT  log_timestamp  FROM `logging`    WHERE log_user_text = 'CodeCat' AND log_user = '21371'  ORDER BY log_timestamp ASC

It is a pretty heavy query;

349238 rows in set (2.24 sec)

(conversation being continued on IRC)

brief update:
@MarcoAurelio and myself have been chatting on IRC

˜/tabbycat 19:16> marostegui: hi
˜/tabbycat 19:16> marostegui: with --group which is the output?
˜/marostegui 19:16> let me see
˜/tabbycat 19:17> I editted the message shortly after my bad sorry
˜/marostegui 19:17> tabbycat: https://phabricator.wikimedia.org/P6057 ?
˜/tabbycat 19:18> okay marostegui so there's no localRenameJobs queued
˜/tabbycat 19:18> there's a maintenance script to use to fix this
˜/tabbycat 19:18> let me check
˜/marostegui 19:18> thanks :)
˜/tabbycat 19:19> docs says we have to wait at least 3 hours before using it because failed jobs will restart themselves
˜/marostegui 19:19> ah
˜/tabbycat 19:19> I'm not sure we've to wait here since there's no job at all
˜/marostegui 19:20> No, i was checking db logs
˜/marostegui 19:20> to look for db errors for enwiktionary
˜/tabbycat 19:21> in any case: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" is the magic command to unstuck the rename job
˜/marostegui 19:21> shall we try it?
/tabbycat 19:21> if legoktm approves
˜/tabbycat 19:22> (due to skipping the 3 hours limit)
˜/marostegui 19:22> let me update the ticket btw

Mentioned in SAL (#wikimedia-operations) [2017-09-28T17:25:51Z] <marostegui> Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985

After more chatting and checks @MarcoAurelio suggested:

mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki --ignorestatus "CodeCat" "Rua"

This looks like unblocked the rename and it went thru enwiktionary and is progressing finely

However renames happening after en.wiktionary are leaving unnatached local accounts. @Legoktm suggest running after the rename has finished:

echo "Rua" >> ~/tmp.txt; mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=metawiki ~/tmp.txt

We should check afterwards why the rename job failed even after running the maintenance script the first time and why this time the rename job left unnatached accounts as that's not something we'd like to happen :)

The command to attach the accounts didn't fix the issue:

root@terbium:~# cat /tmp/rua.txt
Rua
root@terbium:~#  mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=metawiki --userlist /tmp/rua.txt
CentralAuth account attach for: Rua
OK: Rua
[2017-09-28 18:35:13] processed: 1 (4.0/sec); ok: 1 (100.0%); attached: 0 (0.0%); partial: 0 (0.0%); failed: 0 (0.0%); missing: 0 (0.0%);
done.
˜/legoktm 21:23> the good news is that nothing is wrong with the account, just how it is displayed on Special:CentralAuth :)
˜/legoktm 21:23> for whatever reason lu_attached_method is set to NULL and not login or new or whatever
Marostegui closed this task as Resolved.Sep 28 2017, 7:31 PM
Marostegui assigned this task to Legoktm.

Everything is fixed after running the UPDATE below

˜/legoktm 21:28> !log mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";

Thanks a lot @Legoktm and @MarcoAurelio and for helping out and getting this fixed.
I will update the global rename stuck task with all the details about what happened here.

alanajjar added a comment.EditedSep 29 2017, 9:17 AM

Thanks @Legoktm , @MarcoAurelio and @Marostegui

I closed the request in meta