| pmiazga | |
| Jul 9 2025, 2:36 PM |
| F65689714: image.png | |
| Jul 29 2025, 10:46 PM |
It feels like recently, the Global Renames have been getting stuck more often. Just by looking at Phab search (https://phabricator.wikimedia.org/search/query/TC9Iot8bs_IR/#R) we can see there are around 14 support requests in June 2025.
Fixing stuck global renames slowly becomes a chore, where every 3-4 days we need to run a script to unblock another account.
Note: This issue also became a topic during the recent Steward-WMF meeting and was brought to MediaWiki-Platform-Team's attention.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| LocalRenameJob: Continue with other wikis if we mostly succeeded | Bartosz Dziewoński | mediawiki/extensions/CentralAuth | master | +29 -10 |
T400618 came up today and I decided to have a look at the code. https://wikitech.wikimedia.org/wiki/Stuck_global_renames#Debugging_a_stuck_rename has some helpful links to dashboards.
The exception that caused that job to get stuck was:
Error 1213: Deadlock found when trying to get lock; try restarting transaction Function: MediaWiki\User\User::saveSettings Query: UPDATE `actor` SET actor_name = 'Socks' WHERE actor_user = 7481034
We logged this exception at least four times, which is pretty confusing by itself, here's all of the entries I found:
I didn't try to debug the deadlock, my question was instead: Failing jobs are supposed to be retried. Why is this LocalRenameUserJob not getting retried?
This log entry: "Skipping duplicate rename from Editor Socks to Socks" https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.07.28?id=daFSUJgBAlpnrixJ5_Ur suggests that it was retried, but the retried job skipped itself (and didn't trigger the subsequent job on the next wiki), because the rename status was already set to "Done" on commonswiki:
Why was it set to "Done"? Because that exception was thrown after the job code has finished, from a callback scheduled with onTransactionCommitOrIdle() and executed from JobExecutor.php. This is supposed to cause the job to change the rename status to "Failed", but this had no effect because it was already set to "Done" by the job code. It also causes the job not to trigger subsequent jobs, which indeed worked, but left the rename process in a confusing state.
We probably should continue running the jobs when this happens. It means that the local account can be in an inconsistent state (e.g. looking at commonswiki, the 'renameuser' log entry is missing, which should have been inserted at the very end), but if the exception happens after we already committed the main transaction, we can't really do anything about that. We should log some kind of warning, but it is pointless to require a system operator to run a maintenance script to unstuck the job (all the script does is requeue the same kind of job, and we never debug why a problem happened…).
I found similar "Skipping duplicate rename from" log entries in Logstash: https://logstash.wikimedia.org/goto/0ea9a5bfcbc8ab36ef5615c5e702a10b corresponding to almost every recent "Unblock stuck global rename" task: https://phabricator.wikimedia.org/maniphest/query/u6jqBUYRybbg/#R, so I am hopeful that resolving this would resolve most of the future stuck renames.
And by the way, I looked at the actor table entry that was supposed to be updated by the original query that caused the problem, and it is already updated:
mysql:research@dbstore1007.eqiad.wmnet [commonswiki]> select * from actor where actor_user = 7481034; +----------+------------+------------+ | actor_id | actor_user | actor_name | +----------+------------+------------+ | 7486009 | 7481034 | Socks | +----------+------------+------------+
I don't know what updated it, but whatever it was, I would guess that it might have been the cause of the deadlock.
That's probably not it (although it's also a common error). That happens in LocalPageMoveJob, and as far as I can tell, failures there will not interrupt the rename process, they will just leave some user pages under the old titles. Only failures in LocalRenameUserJob interrupt the process.
Change #1174078 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[mediawiki/extensions/CentralAuth@master] LocalRenameJob: Continue with other wikis if we mostly succeeded
Change #1174078 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] LocalRenameJob: Continue with other wikis if we mostly succeeded