Claude points out, reasonably, that taking an hour to write 700GB means we aren't limited by disk write speed anyway. So I think raid6+0 is probably the way to go here unless someone thinks raid6 is bad luck for any reason.

I keep bouncing off this task. Questions that give me pause are:

There's quite a bit of growth here, so even the old storage size doesn't look all that sustainable. Another vote for 0 or 6.

Currently one day of writes are about 700GB

+1

The state of this was odd -- the VM showed state 'active' and was still in the older, smaller size. So somehow Trove's attempt to resize the VM failed even to resize at all. Notably it had an error complaining about a detach operation failing due to being in the 'resize' state. So possibly we got here by doing two resizes in quick succession and the code not being re-entrant... or possibly Trove just dropped the ball.

Here's what a running VM reports:

Sounds fine to me.

We should remove the project entirely, right? I think it only existed to contain the db.

@MoritzMuehlenhoff, I'm hoping you'll have a quick opinion about whether or not it's worth rebooting things to add these flags to libvirt/kvm/cloud-vps. I think I will add the flags to nova either way, but we'd like to avoid the reboots if possible.

all set

In T434128#12410270, @fgiunchedi wrote:

re: the dashboard itself, overcommit ratios are exported as metrics too and don't need to be hardcoded or selected, see the expressions at https://grafana.wikimedia.org/d/fij9xsr/

Looking good. We need to not buy any more of these cursed drives, but in the meantime we're in decent shape. Thank you @BTullis and others!

In T434128#12407096, @Andrew wrote:

https://grafana.wikimedia.org/d/anlf4hf/cloud-capacity-planning

...and now I've added a bit of 'noisy neighbor' info to that panel.

btw I'm trying to provide a top-level dashboard with links here:

I agree that we can probablhy bump overcommit from 2 to 3. Before we do that I'd like to get some visibility into when/if VMs are experiencing CPU blocking due to neighbor VMs; I'd like that to be as simple as measuring steal or load but I'm not convinced it's that straightforward; what do you think?

You are welcome to use our resources for these projects. I have a couple of thoughts, though:

Regarding Magnum: out support is not fantastic but we are in much better shape than we were when I wrote that comment in April. As long as CI folks are fine doing some hands-on cluster maintenance it should be a good option.

In T429387#12394141, @Andrew wrote:

Moving 1043 to from 32k to 16k allocation since it doesn't seem much happier than the others.

Using commands like

I've rebuilt this with a more modern base image and it looks like it's up and healthy now.

I /think/ this is covered by the 'minesweeper' process which was accidentally not running when I opened this.

*bump*

Done. Please re-open if you still hit a limit at 60.

You should be all set. You can access/configure things via https://horizon.wikimedia.org/

Moving 1043 to from 32k to 16k allocation since it doesn't seem much happier than the others.

Loki logs started failing on 2026-09-12; do we think this node was working at all after that day?

OK, I'm learning as a go here :) That "Memory cgroup out of memory: Killed" message is the cgroups working properly: a process in a pod has run over its limits and is being pounded down. So the obsessive restarting is likely the tool itself in futile loop and doesn't indicate a problem with the worker node.

In the attached log from -74, behavior is similar. A container running as k8s-status is getting killed and re-run in the same place over and over. Eventually the noise shifts to a different uid, dannys712-bot, without a lot of info about what happened to the k8s-status container.

"Memory cgroup out of memory" is firing all over the dang place -- I just grepped syslogs throughout toolforge for "Memory cgroup out of memory: Killed" and found it on 24 different nfs workers. In one case, tools-k8s-worker-nfs-24, that message appears 10395 times in the last 18 hours of logging.

I don't think we have a good way of knowing what was scheduled on the node at the time of failure. And, anyway -- is identifying the specific culprit useful? Our kubernetes setup should /never/ permit a user-defined workload to take down a node, however greedy.

Is there a timeline for when instance snapshots could be enabled?

• cloudcephosd1016
• cloudcephosd1017
• cloudcephosd1018
• cloudcephosd1019
• cloudcephosd1020
• cloudcephosd1022
• cloudcephosd1042
• cloudcephosd1043
• cloudcephosd1044 in progress
• cloudcephosd1045
• cloudcephosd1046 (partial)
• cloudcephosd1047

This is down to 8.5G -- not bad!

This is 'fixed' by the attached patch, which is to say: policy doesn't allow snapshots, so the bug is Horizon showing the option in the menu.

K8s eviction thresholds are described as doing what we want. My main concern is that they should already be present with the non-ridiculous cutoff of memory.available<100Mi. Certainly we could lower that and see if the OOM killer stops firing.

Some solutions:

yep

My very first google search turned up this reddit post: "We've seen cases where having a container run the node out of memory can cause Linux to kill unrelated processes, such as the kubelet or even things like the ssh daemon, requiring a note reboot."

I've just added a few more worker nodes; are things working better now?

I'm requesting this so I think that counts as a +1

ftr: I'm creating new worker nodes today in response to T438255. At the moment we seem to be hitting CPU limits vs RAM limits so I'm building with flavor g4.cores24.ram32.disk20.ephem140

The following services are missing logstash logs:

Overnight things seem promising... the drives on 1046 are quite busy but not constantly maxing out like they were for the last trim test.

Attached patch has Keystone in logstash again.

The bullseye deprecation has purged almost all affected VMs. There are probably a few left that were upgraded in place; we'll get alerted if they leak.

Here's a summary dashboard that should tell us most of what we need for hardware purchase planning. Not a lot of growth lately! https://grafana-rw.wikimedia.org/d/anlf4hf/toolforge-and-cloud-capacity-planning?from=now-6h&to=now&timezone=utc&forceLogin=true

suspend, pause and lock are now disallowed for all users. I will be surprised if anyone misses them!

I'm going to try Ben's test, rebuilding osd.334, osd.335, osd.336 and osd.337. For starters I need to drain and destroy them and let the cluster rebalance, which will take ages.

...much later...

On the email thread, Martin asked about 'locking' which turns out to also pause, so I've added that feature to the patch.

Attached patch should disable 'suspend' and 'pause' (both of which prevent live migration.) I've also sent an email to cloud@lists.wikimedia.org just in case someone needs the feature to endure.

In T431307#12346126, @Andrew wrote:

As I feared, this is easy to disable in the API policies but Horizon still displays the option.

As I feared, this is easy to disable in the API policies but Horizon still displays the option.

+1 sgtm

Seems fine to me, but please check back and revert if the cache turns out to not help! +1

The drives in our test cluster seem to be of the same cursed model:

On Monday I will move the proxy to point at the new load balancer, and then deploy this:

The new driver is up and running and in use for >0 real world clusters.

this looks right to me!

This is worth a read, I haven't tried all the suggestions (and certainly haven't tried all of them together):

In T429387#12329114, @BTullis wrote:

I've had a look at this (with LLM assistance) and I think that I found something useful.
The 12 hosts that trip this warning are the 12 hosts fitted with Micron 5xxx PRO SATA SSDs.