Page MenuHomePhabricator

Unblock stuck global rename of Nouminouna to Bnhassin
Closed, ResolvedPublic

Details

Related Gerrit Patches:
mediawiki/extensions/CentralAuth : masterCall LocalRenameUserJob::teardown in FixStuckGlobalRename

Event Timeline

Melos created this task.Apr 22 2017, 3:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Urbanecm triaged this task as High priority.Apr 22 2017, 4:18 PM
Restricted Application added a subscriber: Dereckson. · View Herald TranscriptApr 22 2017, 4:18 PM
DatGuy added a subscriber: DatGuy.Apr 23 2017, 9:26 AM

Just some clarification, this is something with the SQL tables stuck, that they need to be deleted and then the user rerenamed? Not something in the mwconfig, but one for sysadmins?

Just some clarification, this is something with the SQL tables stuck, that they need to be deleted and then the user rerenamed? Not something in the mwconfig, but one for sysadmins?

Yes, you're right. I think everyone with shell access to prod can fix it but I'm not sure.

Just some clarification, this is something with the SQL tables stuck, that they need to be deleted and then the user rerenamed? Not something in the mwconfig, but one for sysadmins?

Yes, you're right. I think everyone with shell access to prod can fix it but I'm not sure.

https://wikitech.wikimedia.org/w/index.php?title=Stuck_global_renames

So I guess users with access to wasat can fix it.

Just some clarification, this is something with the SQL tables stuck, that they need to be deleted and then the user rerenamed? Not something in the mwconfig, but one for sysadmins?

Yes, you're right. I think everyone with shell access to prod can fix it but I'm not sure.

No, this is an undocumented process.

Requeue the job and see if it goes forward?

Dereckson added a comment.EditedApr 23 2017, 3:39 PM

Requeue the job and see if it goes forward?

That's exactly the undocumented part.

Once requeued, it's easy to watch logs for errors, see what was wrong, and if needed, provide a code fix, but the "how to requeue" isn't available in doc.

There is already a Bnhassin account on en.wikipedia:

wasat
$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki Nouminouna Bnhassin
Using Litlok as the renamer.
from: Nouminouna
to: Bnhassin
renamer: Litlok
movepages: 1
suppressredirects:
reason: per [[m:Special:GlobalRenameQueue/request/32575|request]]
ignorestatus:

Starting to run job...
[9ee3ebb2b19df017f5425044] [no req]   DBQueryError from line 1063 of /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: UPDATE  `user` SET user_name = 'Bnhassin',user_touched = '20170423213728' WHERE user_name = 'Nouminouna' AND user_id = '18690097'
Function: RenameuserSQL::rename
Error: 1062 Duplicate entry 'Bnhassin' for key 'user_name' (10.192.16.4)

Backtrace:
#0 /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/Database.php(921): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/Database.php(1547): Database->query(string, string)
#2 /srv/mediawiki/php-1.29.0-wmf.20/extensions/Renameuser/RenameuserSQL.php(190): Database->update(string, array, array, string)
#3 /srv/mediawiki/php-1.29.0-wmf.20/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php(82): RenameuserSQL->rename()
#4 /srv/mediawiki/php-1.29.0-wmf.20/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameJob.php(62): LocalRenameUserJob->doRun(string)
#5 /srv/mediawiki/php-1.29.0-wmf.20/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php(97): LocalRenameJob->run()
#6 /srv/mediawiki/php-1.29.0-wmf.20/maintenance/doMaintenance.php(111): FixStuckGlobalRename->execute()
#7 /srv/mediawiki/php-1.29.0-wmf.20/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php(103): require_once(string)
#8 /srv/mediawiki/multiversion/MWScript.php(99): require_once(string)
#9 {main}

Maybe you can try with --ignorestatus although if it is throwing DB errors
maybe @jcrespo or @Marostegui could have a look.

There isn't any DB error in the meaning "we should do something about database", the issue is in application logic: when this bug was reported, the job started renaming the account, then failed afterwards. The fixStuckGlobalRename calls a regular LocalRenameJob, not expecting this part is already done.

ignorestatus role is to do the task regardless of the status (without it, it must be 'queued' or 'failed' to proceed:

./includes/LocalRenameJob/LocalRenameJob.php
if ( empty( $this->params['ignorestatus'] ) ) {
    if ( $status !== 'queued' && $status !== 'failed' ) {
        LoggerFactory::getInstance( 'CentralAuthRename' )
            ->info( 'skipping duplicate rename from {user}', [
                'user' => $this->params['from'],
                'to' => $this->params['to'],
                'status' => $status,
            ] );
        return true;
    }
}

@Tgr worked on global rename as well IIRC. He might know how to unlock that "failed" local rename. Thanks for your thanks on me porting some docs to Wikitech. Stuck renames started being more frequent and I thought having some docs for all people able to run maintenance scripts would be good. Now, given that fixStuckGlobalRename does not unlock "failed" global renames, we should probably document on that page what to do in those cases as well.

Are you sure --ignorestatus works as you say @Dereckson. I think I remember it forced the job, which is what it seems it's requested here?

Tgr added a comment.Apr 24 2017, 7:45 PM

--ignorestatus means run the script even if the rename status is in progess (which normally means the rename is still ongoing and should not be messed with, but sometimes the rename fails so badly that it can't even update the status to failed). So not applicable here.

Tgr added a comment.Apr 24 2017, 8:00 PM

Both users exist on enwiki. That's not supposed to happen and not fixable by maintenance script.

Please advice if usermerge or local usurpation is desired here.

Dereckson removed Dereckson as the assignee of this task.EditedApr 24 2017, 9:20 PM

Deassigning if someone wants to take care of this this UTC evening.

I'd recommend to tweak the maintenance script and the LocalRenameJob to add a force option to go on with such rename odd case, skipping any row already renamed and updating the other rows.

Tgr renamed this task from Please unblock stuck global rename to Unblock stuck global rename of Nouminouna to Bnhassin.Apr 24 2017, 9:32 PM
Tgr added a comment.Apr 24 2017, 9:50 PM

There are two separate user accounts on enwiki (both 0 edits, one created in 2013, the other in 2014). The rename should never have started in the first place.

Tgr added a comment.EditedApr 25 2017, 2:55 PM

Probably some error prevented the process of user creation for Bnhassin, so the user table did get created but the central user database did not get updated. So the rename was allowed but broke when it reached enwiki since the account already existed there.

(Not sure why Nouminouna does not have logs; it would be a bit too much of a coincidence if both usernames had errors upon creation. FWIW Bnhassin does have logs.)

So we need to delete or rename Bnhassin@enwiki before the rename script can continue.

Tgr added a comment.Apr 25 2017, 2:58 PM

Probably some error prevented the process of user creation for Bnhassin, so the user table did get created but the central user database did not get updated.

OTOH the user account creation is recorded as an autocreation on enwiki, which means a central account must have existed. Maybe there was a previous user rename or merge that was botched somehow?

Tgr added a comment.Apr 30 2017, 9:44 PM

Confirmed that the username Bnhassin does not appear in any table apart from user and logging. Will fix manually with:

BEGIN;
UPDATE user SET user_name = 'Bnhassin~T163622' WHERE user_name = 'Bnhassin';
UPDATE logging SET log_title = 'Bnhassin~T163622' WHERE log_namespace = 2 AND log_title = 'Bnhassin';
COMMIT;

After that, fixStuckGlobalRename.php should make the process resume without problems.

Tgr added a comment.May 1 2017, 6:55 PM

Also needed to

UPDATE localnames SET ln_name = 'Bnhassin~T163622' WHERE ln_wiki = 'enwiki' AND ln_name = 'Bnhassin';

The enwiki rename is successfully finished now, but the next wiki did not start for some reason. No error anywhere as far as I can see. I suppose since the script runs the job manually it should call teardown manually as well?

Tgr closed this task as Resolved.May 1 2017, 7:12 PM
Tgr claimed this task.

Patched the script and restarted the renames.

Change 351181 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/CentralAuth@master] Call LocalRenameUserJob::teardown in FixStuckGlobalRename

https://gerrit.wikimedia.org/r/351181

Change 351181 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Call LocalRenameUserJob::teardown in FixStuckGlobalRename

https://gerrit.wikimedia.org/r/351181

MarcoAurelio moved this task from Backlog to Closed on the GlobalRename board.Jul 24 2017, 8:46 AM
This comment was removed by MarcoAurelio.