Blubber v1.8.0 has been released and contains the new[[ https://doc.wikimedia.org/releng/blubber/examples/06-python-builder.html#installing-specific-versions-of-core-python-packages | fields for setting Python package version constraints ]].

In T418253#11698332, @thcipriani wrote:

Looks like tests pass for https://gerrit.wikimedia.org/r/1250619

In T418813#11675877, @Andrew wrote:

Hey folks, sorry about the not-very-coherent response on this. The bottom line is that compute+storage resources are not an issue, we can definitely provide what you need.

The thing that is in flux our commitment to magnum:

Sounds good to me. Our planned Zuul migration also involves using Magnum for untrusted workloads (see T396936 and related tasks), so yes let's have a cross-team meeting to sync up on expectations. cc @bd808, @thcipriani

Additional napkin math:

I'm being a little squishy with the numbers here, but according to the second highest peak of kube_node_status_allocatable from grafana.cloud.releng.team over the past 3 months, this is where we might want to start with _additional_ CPU/memory quotas (which will be the upper bound of our cluster):

With some moderate refactoring (see https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/551), gitlab-cloud-runner has been successfully provisioned using WMCS Magnum. From the commit message:

@Dzahn After debugging for a bit with @thcipriani I finally just looked at the strace of zookeeper and it appears that zookeeper doesn't have permission to read files beneath /etc/cfssl:

I've verified production access using my new FIDO SSH keys and submitted a patch to remove my old key.

I've added this as a blocker due to the seeming correlation with wmf.10 promotion to group1 today. (See notes in the description.)

@dancy Not quite. We still need the WMCS/trusted runner changes in puppet.

Oh sorry, @Dillon. We collided. I'll set the priority back and leave it as a blocker for historical posterity. Thanks for resolving it.

Deescalating priority and removing this from 1.45.0-wmf.25 train blockers. I'll leave resolution up to you all.

Marking as resolved as I haven't seen the error since deploying the backport yesterday. Please reopen if need be.

I've escalated this issue to a 1.45.0-wmf.25 train blocker. Thanks to everyone already working on it.

This seems to affect commons and wikidata as well. I've seen over 20k errors in the last 10 minutes following group1 promotion of 1.45.0-wmf.25.

Closing this out. @cmassaro please reopen if the latest version of Kokkuri still does not work for you.

In T405119#11313698, @Dzahn wrote:

The other alternative is .. to not try to use TLS. I mean.. none of the other zookeeper servers in WMF prod do it.. as evidenced by the need to add support for it.

And we are not leaving our VM with this traffic.

Ok, great! I'm glad you found a path forward.

Disabling TLS appears to work, so it seems the server does not actually have TLS enabled. :)

@Dzahn Looking at the zuul-web container logs, it seems like the zookeeper connection is failing outright.

@cmassaro try the latest Kokkuri release (2.11.0). It should work with your existing configuration.

Jobs using needs are not getting the dotenv artifacts from kokkuri:setup-variables. Apparently this is a documented side effect of using explicit needs.

In T405118#11230813, @Dzahn wrote:

I tried it and I can confirm using mysql+pymysql gets us past the error.

In T405118#11227582, @thcipriani wrote:

Interesting! This looks like it might be something missing in the executor image. My guess is that we copied upstream, but upstream doesn't use mariadb like we do. @dduvall can you take a look at this?

In T404386#11177989, @bd808 wrote:

In T404386#11177958, @dduvall wrote:

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

No, it is not normal for a project to have 0 Designate zones. There should be svc.$PROJECT.eqiad1.wikimedia.cloud., $PROJECT.eqiad1.wmcloud.org., and $PROJECT.wmcloud.org. zones assigned to the project in Designate.

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

In T404150#11173186, @fnegri wrote:

I asked @Andrew about this, and my understanding is that floating IPs are not required to create Octavia load balancers in OpenStack. But I don't have a full understanding of how Magnum works, so I might be wrong! Can you share more details like tofu code, errors you're getting, etc.?

A spike of these errors occurred during wmf.18 group1 promotion today but strangely all instances of the error were from 1.45.0-wmf.17.

In T392526#10871472, @MoritzMuehlenhoff wrote:

In T392526#10871343, @Scott_French wrote:

I'm also unsure how to resolve the difference in package name - e.g., whether there's some suitable override mechanism (create an empty transitional package?) - or whether there are subtle differences in the packaging configuration that make the result incompatible.

One thing to check is whether these debs are actually arch-dependent, usually most vendor debs are just statically linked, we could try one of them in a bullseye and bookworm container to find out. If so, we could have a single update definition and then use it to sync to bullseye and bookworm.

In T392610#11027111, @20after4 wrote:

SpiderPig is hilarious and awesome. Great job everyone!

In T398873#11026091, @dancy wrote:

Yes, but in the meantime scap prep next would continue to work correctly (assuming at least one prior successful branch cut was merged). As it stands, if`MediaWiki branch and publish WMF single-version image` fails, a subsequent scap prep next will fail.