Any objections to just remove the VM since we moved to (re-)packaging upstream (https://wikitech.wikimedia.org/wiki/Envoy#Building_envoy_for_WMF)?

In T365253#9829691, @MoritzMuehlenhoff wrote:

In T365253#9829677, @elukey wrote:

I checked the dragonfly repo and I have a question about building for bookworm (didn't find it in https://wikitech.wikimedia.org/wiki/Dragonfly) - since we are going to have two os versions, how should we manage the master branch's debian changelog? Namely, should I create a new branch from master for bookworm, or do you prefer another road?

I'd keep it simple and simply move master to bookworm, the legacy packages won't be updated any further and the dragonfly* super nodes will also need to be moved off buster soon.

Not sure if this is the source of it, but full CI runs do fail because the services/mw-videoscaler/staging fails to render

Finally gone...

I just had to deploy machinetranslation for T346638: Rename the envoy's uses_ingress option to sets_sni and noticed container startup times of around 5 minutes (thinking something went totally wrong). I'm still seeing data getting pulled from peopleweb - are the plans to improve this still ongoing?

The ratelimit service has been deployed to staging and prod wikikube clusters.
What's left to be done is to configure cirrus-streaming-updater to use it (see https://wikitech.wikimedia.org/wiki/Ratelimit#Enable/opt_in_to_rate_limiting). From all the values files I'm not sure which components (all?) should be rate limited, so I'd like to leave that change to you @pfischer / @bking / @dcausse. Feel free to send it my way for review/sync with me for the deployment so we can verify everything works as expected.

I see that we run nginx with the default debian nginx.conf which has worker_connections 768; and no worker_rlimit_nofile set. The generic tlsproxy module in puppet uses worker_connections 131072 (no idea where that number comes from) and worker_rlimit_nofile 131072 * 2.

Unfortunately this is actually nginx complaining:

In T353464#9854759, @jijiki wrote:

In T353464#9854751, @CDanis wrote:

In T353464#9854420, @JMeybohm wrote:

In T353464#9845947, @jijiki wrote:

Current status:

As I see it we're currently also still running the ganeti etcd instances in codfw and eqiad which I think does limit the performance of the etcd cluster by quite a bit. Was it a deliberate decision to not remove them?

I think more that we ran out of time to make changes last week. Removing them from the etcd cluster ahead of time seems fine to me, at least.

Any objections to wait for T366204 and T366205 to be completed before we remove the ganeti VMs?

While I you re probably right, I think it is feels slightly easier to just get rid of the old stuff all at once, logistically

In T366465#9854525, @Joe wrote:

I also checked ferm's own @resolve function and it doesn't support SRV records although adding such support wouldn't be too hard either.

In T366465#9854495, @taavi wrote:

Does dnsquery::srv do what you need here?

In T353464#9845947, @jijiki wrote:

Current status:

As I see it we're currently also still running the ganeti etcd instances in codfw and eqiad which I think does limit the performance of the etcd cluster by quite a bit. Was it a deliberate decision to not remove them?

This is basically T287491: Allow to address Kubernetes API servers from NetworkPolicy
IMHO the easiest and less intrusive way to do this with an upstream helm chart is to just add a calico networkpolicy template to the chart (the file could even be prefixed with wmf-) that just creates that one policy. The linked phab task should contain some examples for that.

kubernetes2023 is still cordoned and depooled for additional tests of the move v-lan process

After the reimage I needed to run the following for calico to start up properly:

I've cleared out kubernetes2023 and kubernetes2032 for you to run tests. As the hosts are pooled=inactive and cordoned in k8s all you have to do is to downtime them (which the cookbooks probably do).

@ayounsi I think we could test the rename cookbook together with T350152: Automation to change a server's vlan on the already cordoned kubernetes2023.codfw.wmnet, right?

I might be missing something obvious here, but I've two questions:

• Why add the statsd deployment to the mediawiki chart instead of using a statsd chart, adding a statsd release to mediawiki helmfile.yaml's?
• Why do we need to tunnel statsd trough the local envoy? Can't mediawiki use $namespace.$environment.$release-statsd directly?

Both staging clusters have been migrated to stacked control-planes

For T362408: Migration to containerd and away from docker we're planning to backport containerd from bookworm to bullseye. Maybe it would be feasible to backport runc as well (although this won't help you with T363191: Test if we can avoid ROCm debian packages on k8s nodes ofc.)?

Successfully published image docker-registry.discovery.wmnet/ratelimit:9.0.2-20240503.3fcc360, supporting hot reload of gRPC certs. This should unblock deploying the ratelimit service.

You got me @elukey :-p
For reasons I did not try to understand yet, the mknod cgroup permission is the culprit. Without it, the access() call fails:

Two more data points that don't help at all:

jayme@ml-staging2001:~$ sudo docker exec -it --user 0 k8s_kserve-container_nllb-200-gpu-predictor-00007-deployment-678689d65f-f8xfx_experimental_32d35f31-d95c-48e2-b8c7-f8345eb699e7_0 /usr/bin/python3 -c 'import os; print("real: %d:%d, effective: %d:%d, result: %s" % (os.getuid(),os.getgid(),os.geteuid(),os.getegid(),os.access("/dev/dri/renderD128", os.F_OK)))'
real: 0:0, effective: 0:0, result: False
jayme@ml-staging2001:~$ sudo docker exec -it --user 0 --privileged k8s_kserve-container_nllb-200-gpu-predictor-00007-deployment-678689d65f-f8xfx_experimental_32d35f31-d95c-48e2-b8c7-f8345eb699e7_0 /usr/bin/python3 -c 'import os; print("real: %d:%d, effective: %d:%d, result: %s" % (os.getuid(),os.getgid(),os.geteuid(),os.getegid(),os.access("/dev/dri/renderD128", os.F_OK)))'
real: 0:0, effective: 0:0, result: False

Also worth noting that mediawiki already calls sessionstore via it's envoy sidecar, so we do have telemetry data from prod and we should be able to see the impact there pretty quickly as well: https://grafana-rw.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s?orgId=1&var-datasource=thanos&var-site=eqiad&var-prometheus=k8s&var-app=mediawiki&var-kubernetes_namespace=All&var-destination=sessionstore

It's not clear to me what happened here. The makevm call was unable to detect a puppet run, at that time sudo rules where not present on the machine and puppet did not ran. I've terminated makevm and ran a reimage with more or less the same result but at the time the cookbook was waiting for puppet, there sudo wasn't even installed. install_console was not accessible in both cases, manual root logins via ganeti console did not work as well.
Over night things seem to have cleared up. Even though the cookbook failed puppet did ran, I'm able to login and sudo works (as well as successive puppet runs)...

kubestagemaster2005 got stuck at:

kubestagemaster2004 is done (I messed up the phab ID in the cumin command, so report ended up in https://phabricator.wikimedia.org/T363310#9790605)

I tend to agree, also for sake of alignment of sessionstore with the rest of our services. Unfortunately this feels like the more involved change (A change to sessionstore was somewhat high risk ...) - but I think it's also true that it should not add much to latency.

In T357353#9782050, @EBernhardson wrote:

In T357353#9748141, @sbassett wrote:

In T357353#9748131, @pfischer wrote:

@EBernhardson, according to @JMeybohm there is no way to limit the IP ranges of pod/service/namespace to associated them closely with an application (SUP).

Ok. I know it's hackier, but could that instead be managed via extension config?

I had a ponder, but I'm not sure how yet. With both our app and the mw app servers living in k8s there might be something related we can do with network policies, but I'm not certain.

To be precise here: If the service backing this as well as all consumers are running in the same k8s cluster we could implement network policies that will only allow access from certain workload in the cluster. But I would advice against relying on that because:

• We still have mediawiki appservers in hardware and there will probably be some snowflakes for which I don't know the implications for this
• We won't be able to use this service cross-dc (as we are with all other active/active services), e.g. depooling in an emergency etc. (which would make this a snowflake)

I've not looked in detail (and I probably will not be able to before end of next week) but immediately worrisome to me is the daemonset running with superpowers we decided to not do this with calico for example and instead we distribute the cni plugin via debian packages - would that potentially be an option as well or has it been considered?

I did deploy the cert-manager changes to aux, @brouberol did dse and @klausman will take care of ml clusters, thanks all

In T362938#9774724, @Jhancock.wm wrote:

Forgot I left it there. All yours now!

Pushing both branches worked now, thanks!

@akosiaris maybe you recall if there was a deliberate decision not to use service mesh for kask/session store?

In T362938#9764496, @Jhancock.wm wrote:

@JMeybohm papaul helped me identify the missing disk. I replaced it with a compatible drive. please let me know if that fixed the issue. Thanks.

Thank you!, almost there. It now fails with:

In T345823#9766568, @cmooney wrote:

In T345823#9763845, @JMeybohm wrote:

We decided during migration of production to a bigger Pod IP space that this will not be necessary for staging and it actually is not. The issue there (as we figured later) that the IP space is split into /26 blocks, effectively limiting the cluster size to 4 nodes (including control-plane). The change to the IP block size was made to overcome this limitation without the need of changing the Pod IP space (and therefore having to reconfigure that in various places).

Ok cool, and yep makes perfect sense in staging we won't have a high number of pods. Plan sounds good so, I just wanted to make sure we weren't being too conservative with the allocations.

Regarding the current limit of 50 routes announced max from each host I think that is still ok? We're still slightly confused about how it tripped, seems like during the change the host briefly sent more than we expected? But should be ok in general?

In T363996#9763646, @MoritzMuehlenhoff wrote:

This certificate doesn't show up anywhere in certificate.manifests.d for cergen, though?

In T345823#9763715, @cmooney wrote:

Not sure if it might be worth taking a step back and weighing up what's happening here?

As I understand it there is a /24 IPv4 allocation for POD IPs for this cluster, and with the current IP block size at /26 that only provides 4 blocks?

Without knowing the details there are probably two ways to deal with this:

1. Allocate a large block than a /24 for such use, providing more /26 blocks that can be used
2. Keep the /24 overall allocation as it is, but make the IP blocks smaller so there are more overall (/28, /29, /30 or whatever)

From a netops perspective we are relatively agnostic here, however given this is private IP space we have some flexibility. We definitely should try to avoid making any decisions that will potentially bite us down the road. Are we potentially putting too much of a limit on the number of potential PODs per host if we use a block size of /28 or less? Might it be better to keep those block allocations at /26 to allow for growth?

Should be fine either way, but just want to raise the question. We also need to size the 'prefix limit' on our network gear appropriately, current value of 50 should be ok for /28, but we may want to adjust up if using /30 or /32.

staging-eqiad has been migrated to /28 blocks as well

In T362938#9761317, @jnuche wrote:

Scap failed to connect to this host today during the MediaWiki train while trying to preload the MW image:
15:08:17 /usr/bin/sudo /usr/local/sbin/mediawiki-image-download 2024-05-01-150512-publish (ran as mwdeploy@mw2382.codfw.wmnet) returned [255]: ssh: connect to host mw2382.codfw.wmnet port 22: Connection timed out

Would it be possibly to remove it temporarily from the list of K8s workers while work is done on it?