@dancy Not quite. We still need the WMCS/trusted runner changes in puppet.

Oh sorry, @Dillon. We collided. I'll set the priority back and leave it as a blocker for historical posterity. Thanks for resolving it.

Deescalating priority and removing this from 1.45.0-wmf.25 train blockers. I'll leave resolution up to you all.

Marking as resolved as I haven't seen the error since deploying the backport yesterday. Please reopen if need be.

I've escalated this issue to a 1.45.0-wmf.25 train blocker. Thanks to everyone already working on it.

This seems to affect commons and wikidata as well. I've seen over 20k errors in the last 10 minutes following group1 promotion of 1.45.0-wmf.25.

Closing this out. @cmassaro please reopen if the latest version of Kokkuri still does not work for you.

In T405119#11313698, @Dzahn wrote:

The other alternative is .. to not try to use TLS. I mean.. none of the other zookeeper servers in WMF prod do it.. as evidenced by the need to add support for it.

And we are not leaving our VM with this traffic.

Ok, great! I'm glad you found a path forward.

Disabling TLS appears to work, so it seems the server does not actually have TLS enabled. :)

@Dzahn Looking at the zuul-web container logs, it seems like the zookeeper connection is failing outright.

@cmassaro try the latest Kokkuri release (2.11.0). It should work with your existing configuration.

Jobs using needs are not getting the dotenv artifacts from kokkuri:setup-variables. Apparently this is a documented side effect of using explicit needs.

In T405118#11230813, @Dzahn wrote:

I tried it and I can confirm using mysql+pymysql gets us past the error.

In T405118#11227582, @thcipriani wrote:

Interesting! This looks like it might be something missing in the executor image. My guess is that we copied upstream, but upstream doesn't use mariadb like we do. @dduvall can you take a look at this?

In T404386#11177989, @bd808 wrote:

In T404386#11177958, @dduvall wrote:

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

No, it is not normal for a project to have 0 Designate zones. There should be svc.$PROJECT.eqiad1.wikimedia.cloud., $PROJECT.eqiad1.wmcloud.org., and $PROJECT.wmcloud.org. zones assigned to the project in Designate.

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

In T404150#11173186, @fnegri wrote:

I asked @Andrew about this, and my understanding is that floating IPs are not required to create Octavia load balancers in OpenStack. But I don't have a full understanding of how Magnum works, so I might be wrong! Can you share more details like tofu code, errors you're getting, etc.?

A spike of these errors occurred during wmf.18 group1 promotion today but strangely all instances of the error were from 1.45.0-wmf.17.

In T392526#10871472, @MoritzMuehlenhoff wrote:

In T392526#10871343, @Scott_French wrote:

I'm also unsure how to resolve the difference in package name - e.g., whether there's some suitable override mechanism (create an empty transitional package?) - or whether there are subtle differences in the packaging configuration that make the result incompatible.

One thing to check is whether these debs are actually arch-dependent, usually most vendor debs are just statically linked, we could try one of them in a bullseye and bookworm container to find out. If so, we could have a single update definition and then use it to sync to bullseye and bookworm.

In T392610#11027111, @20after4 wrote:

SpiderPig is hilarious and awesome. Great job everyone!

In T398873#11026091, @dancy wrote:

Yes, but in the meantime scap prep next would continue to work correctly (assuming at least one prior successful branch cut was merged). As it stands, if`MediaWiki branch and publish WMF single-version image` fails, a subsequent scap prep next will fail.

In T398873#11026078, @dancy wrote:

@dduvall I'd like to see MediaWiki branch and publish WMF single-version image changed so that instead of destroying and recreating the wmf/next branch each time it runs, it updates wmf/next if it already exists. This means being able to handle added/dropped extensions.

In T398873#10986500, @dancy wrote:

If CI fails (which happens about 10% of the time), we're left with an unusable wmf/next branch until the next run

@bd808 Kokkuri 2.8.0 will include the digest in the image ref. See if that solves your issue.

@Dzahn the WMF based production images for Zuul and Nodepool have been built and published to our registry. I'll post a summary about how we're managing them in T396245: Build zuul images for production tomorrow, but here are the latest image refs by service:

Looking at the above results, I believe that most of the functionality being served by PipelineLib could potentially be served by docker buildx bake (in conjunction w/ buildkitd and Blubber). Docker bake can build multiple sets of targets/contexts/configs simultaneously and even export the results as generic artifacts (to serve the one case that is using copy).

PipelineLib actions in use, according to codesearch results of 27 Gerrit hosted projects that include a .pipeline/config.yaml file.

I refactored the blubber.yaml that @dancy had written back when we were experimenting with a Zuul setup for GitLab and created a wmf/12.0.0 branch.

Spotted this today as well, following wmf.4 promotion to all wikis.

@thcipriani is this still a blocker or are we good for group1/all wiki promotion today?

In T390251#10759166, @dduvall wrote:

In T390251#10751229, @elukey wrote:

serializes only layers per push. Multiple pushes can still happen simultaneously and IIUC scap does do that. Maybe we could have a flag in scap like e.g. --image-push-concurrency=1 to at least verify/rule out this hypothesis? It would slow down deployment for a couple of weeks but would give a strong signal to guide us better into resolving this.

+1 I like this, @dancy @dduvall what do you think about it?

As a short-term mitigation, it seems reasonable, and serializing the image pushes in build-images.py should be straightforward.

In T390251#10751229, @elukey wrote:

serializes only layers per push. Multiple pushes can still happen simultaneously and IIUC scap does do that. Maybe we could have a flag in scap like e.g. --image-push-concurrency=1 to at least verify/rule out this hypothesis? It would slow down deployment for a couple of weeks but would give a strong signal to guide us better into resolving this.

+1 I like this, @dancy @dduvall what do you think about it?

Looks to have been introduced in:

Removing this task as a blocker as the errors only occurred during a short-ish window, occurred for wmf.24 as well as wmf.25, and only for internal wikis.

Closed as a duplicate that, while ongoing, is not strictly a train blocker.

Other possibly relevant discussions around this issue.

In T390251#10744699, @dduvall wrote:

Also, I wonder if there's a way we can force monolithic uploads?

Also, I wonder if there's a way we can force monolithic uploads?

In T390251#10744640, @dancy wrote:

In the case of uploads, here is the bad sequence:

• The client (e.g. dockerd) issues POST /v2/<repo>/blob/uploads/ to initiate an upload. This returns a new URL for subsequent operations (hereafter called the upload URL)
• The client issues a PATCH to the upload URL to transmit the data.
• The client issues a PUT to the upload URL to finalize the upload. This is where a 404 is sometimes returned by the registry (basically saying that it doesn't know about this upload). A 404 is more likely to be seen if a prior upload was large (i.e, if the replicator is busy). Retrying this PUT does eventually succeed.

In this case it's not the content of the upload that hasn't made it to the replica, but the existence of the upload state itself.

In T390251#10744390, @dancy wrote:

At this point we have two problems.

• Large image pushes are now unreliable (this seems new for mediawiki deployments). No workaround proposed yet.

The Zuul dashboard is available at https://zuul-dev.wmcloud.org/tenants

In T391374#10740144, @hashar wrote:

@dduvall excellent!

The event stream permission is probably good enough, it does not grant any specific access beside the ability to receive events and we have multiple bots on WMCS using that same setup. As long as the user is not granted more permission, it can't do much. Setting up a Gerrit + repos + config might add a bit of a burden, then if you can reuse an existing setup that let gives us a great playground \o/

@hashar FYI I've set up Zuul and friends on zuul-1001.zuul3.eqiad1.wikimedia.cloud using https://opendev.org/zuul/zuul/src/branch/master/doc/source/examples/docker-compose.yaml

In T389499#10676584, @Scott_French wrote:

So, a possibly adequate analogy for the relationship between image "kind" (a new name for an existing concept that did not previously have a name) and image "flavour" (an existing name for an existing concept) would be that between a class definition and the specific set of constructor arguments that produce a concrete instantiation.

In T388769#10644219, @bd808 wrote:

The apt config and implementation is also Debian-based base image specific. Alpine base images would need an apk config and implementation. Red Hat-based base images would need a yum (or dnf?) config and implementation. Arch uses pacman. OpenSUSE uses zypper. I am not aware of any unifying abstraction over the various distro specific package managers that would simplify this readily.

Thanks for pointing out that this isn't a bug, @bd808.

In T387927#10614474, @dancy wrote:

I agree with running a daily timer and trying to spread the knowledge about the availability of scap clean-images to quickly recover space in unusual circumstances.

The scap clean-images implementation has been merged. I plan on doing a release early next week. Sample behavior from train-dev: