I opened this related task: T330194

Renaming branch in another task.

Fixes were deployed in scap 4.37.0

I will add some extra output to clarify what is happening (checking for dependencies of the dependency).

MR above modifies the depends_on check to request dependecies for a change_id in a specific project & branch to avoid the multiple changes error. This functionality existed previously but was removed in a refactor (oops), so I added a comment to make sure it doesn't happen again, but maybe best to add a test for this.

In T323277#8597192, @Urbanecm wrote:

In T323277#8596212, @Tgr wrote:

Or maybe just leave it to the user to backport the dependencies with separate scap backport calls, and ask for manual confirmation that that happened (or check via git but that seems more fragile).

AFAIK, once a change has a dependency with a backport, scap backport fails even with a single change specified. I had to fetch said change manually and do a scap sync-world instead.

Is this resolved now?

I did some more investigation and I think the deployment couldn't continue since it was searching for a commit with "Change-Id: I39ab54ca6e9f9a6a58b59cdd6feea07638fc908f", but the commit for the patch in question had "Change-ID: I39ab54ca6e9f9a6a58b59cdd6feea07638fc908f". I thought the Change-Id line was case sensitive, but it seems not since the patch was accepted into gerrit, so I'll change the search to reflect that.

So far I haven't reproduced this issue in our test environment. If it happens again please update with a copy of the output up until it breaks if you can.

This was done, but disabled due to poor performance. A summary of results is in this commit message: https://gitlab.wikimedia.org/repos/releng/kokkuri/-/commit/1dfd3a0d6582fc284200c4db34661de06e5a5c78

what about having it do everything except sync if it's not deployed anywhere?

In T314189#8218522, @kostajh wrote:

Some options I can think of:

• rollback to group0 and group1 wikis to wmf.27, to help determine if there are code changes in core and/or extensions/skins that are contributing. The ElasticSearch changes are surely a contributor to what we see on Special:Homepage but shouldn't be responsble for the overall increase in performance metrics, AIUI
• rollback to php 7.2 for some period of time to observe what happens with https://performance.wikimedia.org/#!/week and other charts
• rollback to stage 3 of T279664: Progressive Multi-DC roll out, in order to rule out that as a contributing factor

In T314189#8217470, @Ladsgroup wrote:

There is a general increase in every performance metric it seems: https://performance.wikimedia.org/#!/week

We probably need to look at flame graphs. php 7.4 usually shouldn't cause this, it's demonstrably faster.

https://gerrit.wikimedia.org/r/815277
Rolled back all wikis to wmf.19

👍 thanks @Dzahn, sorry for the misunderstanding

@Dzahn I was only commenting to provide information that in order to merge the patch in question, a different version would be needed, and finding out what version that would be if we were to do so, not to make a decision on whether or how to upgrade.

I saw a question on IRC about which version is needed. According to the docs, https://pypi.org/project/ruamel.yaml/ version 0.16.0 or greater would work

checking /var/log/php7.2-fpm on mw1369 I see a restart around deployment time: