Page MenuHomePhabricator

Global renames aren't being processed on beta cluster
Closed, ResolvedPublic

Description

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

As for DannyS712_test I looked at Logshash for errors that day but apparently there's none. I'll run the script to unblock it manually.

No queued jobs apparently either:

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=enwiki --type LocalRenameUserJob
0

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=enwiki --type RenameUserJob
0
maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki '"DannyS712 tes"' '"DannyS712_test"'

"DannyS712 test" does not have a rename in progress on this wiki.

Tried to login to DannyS712 test - "Your account is currently being renamed or merged. View the status." (links to https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/DannyS712_test)

As for ~riley:

maurelio@deployment-deploy01:~$ mwscript showJobs.php --wiki=arwiki

0

I wonder if that script is working.

No Logstash entries for CentralAuthRename or Renameuser channel on Logstash-Beta (looking from 2019-01-01 to date): https://kibana4.wmflabs.org/goto/370bf075fc5f620d681f50194a2f7587

Not sure what's going on.

I see errors to the level of EMERG for deployment-cpjobqueue for today at https://logstash-beta.wmflabs.org/goto/2a6649ad504b7f0833c9db556aab78b6 but I'm not sure that'd have anything to do with this.

@Krenair Could you please take a look?

Ping to @Legoktm as well as I think he's very familiar with CentralAuth/RenameUser.

DannyS712 moved this task from Unsorted to Reports on the User-DannyS712 board.Dec 23 2019, 11:44 AM

For debug: All events from 21 to 22 December 2019. Bunch of cpjobqueue errors: https://logstash-beta.wmflabs.org/goto/18db76a89c538107dc2ad7c8a11e0559

I tried to unblock ~riley's rename using the script as well, but I get the same errors as in T241294#5759652. I tried to rename a spambot account and it has even refused to start. I suspect this is jobqueue being broken. I've filed a task for that.

I performed a rename yesterday after various instances and services restarts, and other maintenance (cfr. T241462). It was a brand new spambot account with no edits, and just three wikis attached. It worked but took ca. 6 minutes to complete. Certainly not a normal execution time for production where such renames would take seconds to complete. I'll see if I can restart those renames with the fixStuckGlobalRename.php script now that the JobQueue seems to be processing (some?) data again.

Nah, it does not work:

maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki '"DannyS712 tes"' '"DannyS712_test"'

"DannyS712 test" does not have a rename in progress on this wiki.


maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki '"DannyS712 tes"' '"DannyS712_test"' --ignorestatus

"DannyS712 test" does not have a rename in progress on this wiki.

but it is in the centralauth.renameuser_queue table:

MariaDB [centralauth]> select * from renameuser_status;
+----------------+-------------------------+----------------+-----------+
| ru_oldname     | ru_newname              | ru_wiki        | ru_status |
+----------------+-------------------------+----------------+-----------+
| DannyS712 tes  | DannyS712 test          | enwiki         | queued    |
| DannyS712 tes  | DannyS712 test          | loginwiki      | queued    |
| DannyS712 tes  | DannyS712 test          | metawiki       | queued    |
| HoseaSeverson6 | Random spammer username | deploymentwiki | queued    |
| Riley Huntley  | ~riley                  | arwiki         | queued    |
| Riley Huntley  | ~riley                  | commonswiki    | queued    |
| Riley Huntley  | ~riley                  | deploymentwiki | queued    |
| Riley Huntley  | ~riley                  | dewiki         | queued    |
| Riley Huntley  | ~riley                  | enwiki         | queued    |
| Riley Huntley  | ~riley                  | enwikibooks    | queued    |
| Riley Huntley  | ~riley                  | enwikinews     | queued    |
| Riley Huntley  | ~riley                  | enwikiquote    | queued    |
| Riley Huntley  | ~riley                  | enwikisource   | queued    |
| Riley Huntley  | ~riley                  | enwikiversity  | queued    |
| Riley Huntley  | ~riley                  | enwiktionary   | queued    |
| Riley Huntley  | ~riley                  | eswiki         | queued    |
| Riley Huntley  | ~riley                  | loginwiki      | queued    |
| Riley Huntley  | ~riley                  | metawiki       | queued    |
| Riley Huntley  | ~riley                  | simplewiki     | queued    |
| Riley Huntley  | ~riley                  | testwiki       | queued    |
+----------------+-------------------------+----------------+-----------+
20 rows in set (0.00 sec)

I see [[ https://phabricator.wikimedia.org/diffusion/ECAU/browse/master/maintenance/fixStuckGlobalRename.php | fixStuckGlobalRename.php ]] tries to use DB_REPLICA to work. I was wondering if we could add an option so it could use DB_MASTER (such as --use-master) instead and see if that resolves the issue?

Another explanation is that those jobs were lost, thus not sure how to introduce them in the queue again.

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:20:59Z] <Urbanecm> urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki 'DannyS712 tes' 'DannyS712 test' (T241294)

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:23:50Z] <Urbanecm> urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=arwiki --logwiki=deploymentwiki 'Riley Huntley' '~riley' (T241294)

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

This comment was removed by Urbanecm.

Mentioned in SAL (#wikimedia-releng) [2019-12-28T14:58:10Z] <hauskatze> maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'HoseaSeverson6' 'Random spammer username' | T241294

There's now no stuck global renames anymore; but the issue with renames taking forever to perform still remains to be answered.

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

Thank you. Using single quotes and no underscores indeed seems to have let me unblock the third remaining stuck global rename.

@MarcoAurelio Found the issue. fixStuckGlobalRename.php requires spaces, the MediaWiki doctrine that spaces equals underscores is not true in this case. Run the command and Danny's rename is fully processed, and riley's rename started to move slowly. It's still a question why renames are processed slowly.

So I took a look at https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/DannyS712_test - why does it say "admin-merged" account? Is that because of the script?

Yup. The data isn't recoverable if renamed becomes stuck.

Now that the joq queue should be running, I've tested https://deployment.wikipedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/T241294 to check that this is working

DannyS712 closed this task as Resolved.Jan 9 2020, 5:03 AM

Its working