@matmarex I wasn't expecting anyone to go through every single duplicated translation. Thank you so much for taking the time to do that! :D

Thank you @Dreamy_Jazz . I can close this next week when wmf.27 is fully deployed.

With wmf.26 deployed to all wikis we still see quite a few logs for this kind of warnings. Right now they are among the most frequent we can see in the new errors board (after removing the three filters for T361695):

Noting here that after analyzing the issue with @dancy and @jeena. there's a few extra situations in backport that we want to control. This will also include some more refactoring of existing code. I still need to create a task to document and track the necessary work for this.

Nope, this is the one I wanted to take. Apologies again, I need coffee...

Ooops, sry, picked the wrong task

Thanks @Dreamy_Jazz!

The exception has actually been happening with different offset indices for at least a week, e.g. on the 2024-03-28 (Thursday) last week during the wmf.24 train. This aligns better with the deployment of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1004792

After train deploy to group2, elwiki and cewiki are also affected by some duplicated translation. I don't see any other wikis affected so far.

@Dreamy_Jazz thank a lot for looking into it so quickly.

@TheDJ Thank you, then the impact of the issue is probably limited. I'm lowering the priority accordingly.

Making a note here for T361695. If the error rate climbs up once we deploy to group2 it can possibly require a revert of the patch.

@TheresNoTime I was just about to reduce the priority, thx :)

Fix will go out with the next scap release

@taavi kiitos :)

Thanks for taking a look @taavi! Can you also kill the service?

Release Jenkins instances have now been upgraded to 2.440.2. An update of some configuration and the matrix-auth plugin was also required: https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/57

The fix from https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/57 has fixed the issue. Both releases Jenkins instances have now been updated to 2.440.2 and look healthy.

This should be complete now, including notifying failing patches during the weekly train branch cut.

Looked into it and this was a false positive.

I've tried reinstalling scap and that has re-added the missing dependency files:

In T358237#9611992, @Dzahn wrote:

zuul has now succesfully been deployed to this machine by @jnuche :)

Testing message fix. Changing priority manually to allow for another notification to get through.

The patch paths mentioned in the scap commands above are wrong. They should read: /srv/patches/1.42.0-wmf.22/core/01-T355538.patch.

The problem seems fixed in prod now. Thank you everyone your your help with this.

This one currently fails: https://pl.wikiquote.org/api/rest_v1/page/html/Baldur's_Gate/565992?redirect=false

@daniel thanks, created a version for 1.42.0-wmf.21 too.

https://gerrit.wikimedia.org/r/1009500 didn't solve the issue: https://grafana.wikimedia.org/goto/KZMbFoASz?orgId=1

@Jgiannelos @Jdforrester-WMF I've created a backport patch to unblock the train: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009500. But it would still need a pair of eyes before I'm comfortable backporting it

Ah, @Jdforrester-WMF came to the rescue, thank you :)

@daniel 👋 Ok to go ahead with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009237?

Both instances have now been updated.

I've updated the releases instance.

@Urbanecm_WMF thanks!

@daniel @Urbanecm_WMF We need a backport patch to unblock 1.42.0-wmf.21.

Releases instance has affected plugins:

• https://plugins.jenkins.io/trilead-api
• https://plugins.jenkins.io/htmlpublisher

Do we have an idea of how bad the impact for group1 users is? Does this warrant a rollback?

Thanks @matmarex for the +1 :)

In T359290#9605695, @Kizule wrote:

In T359290#9605680, @jnuche wrote:

I've just created a backport patch to unblock 1.42.0-wmf.21. Could someone take a look?

I guess it would be the safest option to schedule it for today's "UTC afternoon backport window".

I've just created a backport patch to unblock 1.42.0-wmf.21. Could someone take a look?

I was too hasty closing this, maybe more work is required on master. Sorry about the noise.

@Jdlrobson @SToyofuku-WMF Thank you for the fix! It's deployed now.