Any job should really try to be as idempotent as possible, otherwise it will always be risky.

If the hangup is from RunSingleJob.php, but nothing was logged, I wonder if the job went through the first time, but changeprop lost the connection and thus retried, which could breakage if the job cannot run twice.

Looking at scap logs, it almost certainly comes from deployments terminating and recreating mw-jobrunner pods, thus killing the jobs in flight.

In T402435#11105193, @aaron wrote:

Any job should really try to be as idempotent as possible, otherwise it will always be risky.

In principle, of course it should; in practice, the rename code is spread across core, CentralAuth and random extensions, and it is difficult to make it better.

In T402435#11105755, @Clement_Goubert wrote:

Looking at scap logs, it almost certainly comes from deployments terminating and recreating mw-jobrunner pods, thus killing the jobs in flight.

You know, I joked that this looks almost as if someone unplugged the servers, I didn't realize how close to true that was… :) I'm glad that at least there's an explanation.

Would it be difficult to make the deployment process depool the pods or something, and give the in-progress jobs some time to finish, before terminating them?

Global rename jobs are mostly idempotent - if they abort and someone re-runs them, the end result is almost always the same as if they didn't abort in the first place. They do require someone to re-run them though, and the user is locked out in the meantime. So it's not great when they get silently discarded.

Serviceops backlog triaging here, @Tgr could you confirm if we can close this task?

In T402435#11529157, @Clement_Goubert wrote:

@Tgr Does the work done in T402830: Global rename jobs should use a lock rather than storing "in progress" state in the database make the jobs rerun without manual intervention?

I was somewhat wrong earlier - the job runner tries the rename three times, and I don't think scap killing the pod is special in that respect; the job will be retried because it's not removed from the job queue until it succeeds or the retry count is reached.

T402830 fixed an issue that caused retries to fail when the initial process was killed abruptly. Whether that was the only such issue, I think we'll just have to wait and see.