As for DannyS712_test I looked at Logshash for errors that day but apparently there's none. I'll run the script to unblock it manually.

No queued jobs apparently either:

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=enwiki --type LocalRenameUserJob
0

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=enwiki --type RenameUserJob
0

Tried to login to DannyS712 test - "Your account is currently being renamed or merged. View the status." (links to https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/DannyS712_test)

As for ~riley:

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=arwiki

0

I wonder if that script is working.

No Logstash entries for CentralAuthRename or Renameuser channel on Logstash-Beta (looking from 2019-01-01 to date): https://kibana4.wmflabs.org/goto/370bf075fc5f620d681f50194a2f7587

Not sure what's going on.

I see errors to the level of EMERG for deployment-cpjobqueue for today at https://logstash-beta.wmflabs.org/goto/2a6649ad504b7f0833c9db556aab78b6 but I'm not sure that'd have anything to do with this.

@Krenair Could you please take a look?

Ping to @Legoktm as well as I think he's very familiar with CentralAuth/RenameUser.

For debug: All events from 21 to 22 December 2019. Bunch of cpjobqueue errors: https://logstash-beta.wmflabs.org/goto/18db76a89c538107dc2ad7c8a11e0559

I tried to unblock ~riley's rename using the script as well, but I get the same errors as in T241294#5759652. I tried to rename a spambot account and it has even refused to start. I suspect this is jobqueue being broken. I've filed a task for that.

I performed a rename yesterday after various instances and services restarts, and other maintenance (cfr. T241462). It was a brand new spambot account with no edits, and just three wikis attached. It worked but took ca. 6 minutes to complete. Certainly not a normal execution time for production where such renames would take seconds to complete. I'll see if I can restart those renames with the fixStuckGlobalRename.php script now that the JobQueue seems to be processing (some?) data again.

Nah, it does not work:

maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki '"DannyS712 tes"' '"DannyS712_test"'

"DannyS712 test" does not have a rename in progress on this wiki.


maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki '"DannyS712 tes"' '"DannyS712_test"' --ignorestatus

"DannyS712 test" does not have a rename in progress on this wiki.

but it is in the centralauth.renameuser_queue table:

MariaDB [centralauth]> select * from renameuser_status;
+----------------+-------------------------+----------------+-----------+
| ru_oldname     | ru_newname              | ru_wiki        | ru_status |
+----------------+-------------------------+----------------+-----------+
| DannyS712 tes  | DannyS712 test          | enwiki         | queued    |
| DannyS712 tes  | DannyS712 test          | loginwiki      | queued    |
| DannyS712 tes  | DannyS712 test          | metawiki       | queued    |
| HoseaSeverson6 | Random spammer username | deploymentwiki | queued    |
| Riley Huntley  | ~riley                  | arwiki         | queued    |
| Riley Huntley  | ~riley                  | commonswiki    | queued    |
| Riley Huntley  | ~riley                  | deploymentwiki | queued    |
| Riley Huntley  | ~riley                  | dewiki         | queued    |
| Riley Huntley  | ~riley                  | enwiki         | queued    |
| Riley Huntley  | ~riley                  | enwikibooks    | queued    |
| Riley Huntley  | ~riley                  | enwikinews     | queued    |
| Riley Huntley  | ~riley                  | enwikiquote    | queued    |
| Riley Huntley  | ~riley                  | enwikisource   | queued    |
| Riley Huntley  | ~riley                  | enwikiversity  | queued    |
| Riley Huntley  | ~riley                  | enwiktionary   | queued    |
| Riley Huntley  | ~riley                  | eswiki         | queued    |
| Riley Huntley  | ~riley                  | loginwiki      | queued    |
| Riley Huntley  | ~riley                  | metawiki       | queued    |
| Riley Huntley  | ~riley                  | simplewiki     | queued    |
| Riley Huntley  | ~riley                  | testwiki       | queued    |
+----------------+-------------------------+----------------+-----------+
20 rows in set (0.00 sec)

I see [[ https://phabricator.wikimedia.org/diffusion/ECAU/browse/master/maintenance/fixStuckGlobalRename.php | fixStuckGlobalRename.php ]] tries to use DB_REPLICA to work. I was wondering if we could add an option so it could use DB_MASTER (such as --use-master) instead and see if that resolves the issue?

Another explanation is that those jobs were lost, thus not sure how to introduce them in the queue again.

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:20:59Z] <Urbanecm> urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki 'DannyS712 tes' 'DannyS712 test' (T241294)

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:23:50Z] <Urbanecm> urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=arwiki --logwiki=deploymentwiki 'Riley Huntley' '~riley' (T241294)

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:58:10Z] <hauskatze> maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'HoseaSeverson6' 'Random spammer username' | T241294

There's now no stuck global renames anymore; but the issue with renames taking forever to perform still remains to be answered.

In T241294#5765194, @Urbanecm wrote:

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

Thank you. Using single quotes and no underscores indeed seems to have let me unblock the third remaining stuck global rename.

In T241294#5765194, @Urbanecm wrote:

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

So I took a look at https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/DannyS712_test - why does it say "admin-merged" account? Is that because of the script?

Yup. The data isn't recoverable if renamed becomes stuck.

Now that the joq queue should be running, I've tested https://deployment.wikipedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/T241294 to check that this is working