JFYI we can now proceed with cloudcephosd1052 too

In T399180#11432250, @cmooney wrote:

In T399180#11432052, @fgiunchedi wrote:

I think the easiest would be to:

• Remove the spurious enp13s0f1np1 config, run puppet to verify no other changes will be applied
• Make sure the ifupdown config matches e.g. cloudcephosd1050, modulo addresses
• Reboot the host and verify addresses/interfaces come up as expected

Yeah I think this should be ok.

Checking the kube-state-metrics container logs I get the following:

Something I noticed is that kube-state-metrics as a whole has been occasionally and lately failing scrapes from prometheus: (long url, toolforge.org isn't allowed on w.wiki) https://prometheus.svc.toolforge.org/tools/graph?g0.expr=sum_over_time(up%7Bjob%3D%22k8s-kube-state-metrics%22%7D%5B1h%5D)%2F%20count_over_time(up%7Bjob%3D%22k8s-kube-state-metrics%22%7D%5B1h%5D)&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=4w&g0.end_input=2025-12-12%2016%3A54%3A05&g0.moment_input=2025-12-12%2016%3A54%3A05

I took a look at why cloudcephosd1052 still has second nic up, currently:

This is done! I'll followup with an announcement to sre@

I dug into this a little, currently:

Availability as seen by network probes:

ALSO: make sure grafana's default is UTC on new dashboards

I'm aware there is/was work going on on thanos/titan in T410152: Disk space saturation (/srv) on Titan hosts and perhaps related

As to effectively do the audit, we can adapt search-grafana-dashboards.js from https://wikitech.wikimedia.org/wiki/Grafana#Search/audit_metrics_usage_across_dashboards

I briefly looked the clouddumps1002 downtime from yesterday, and of course there was ~30m downtime for dumps.w.o since 1002 serves those:

Something else to note: the alerts are deployed in eqiad only, not codfw

Leaving a suggestion here for a workaround for the record: while having a native pg facility to detect password changes would be optimal; I think what we could do is write the (hashed, possibly salted with a salt we control) passwords on the filesystem (e.g. one per file). Then use the following logic:

• if the password file doesn't exist, the user and its password needs to be created in pg and the fs
• if the password file does exist, compare its value with the current puppet password. If they differ then update pg and the fs. If they don't then there's nothing to do.

Something I wanted to add: I'm not very familiar with that part of the puppet codebase though I was wondering if we can start referring to the networks and their attributes by name. It will enable us to have puppet code e.g. "give me the subnets for VXLAN/IPv6-dualstack, either v4 or v6 or both", in other words get to more understandable and manageable code/understanding. Ditto for other attributes/properties for a given network like its gateway, etc. Let me know what you think!

Looks good to me! Definitely good to refactor these bits

In T410989#11405248, @cmooney wrote:

Thinking it through what is probably best:

1. We disable the switch interfaces terminating these second ports now
2. We re-run the PuppetDB import script for these hosts to update their Netbox interface based on current status
3. We ask DC-Ops to remove all the cables on site, deleting each one in Netbox as they go

I can take care of 1 and 2 I think, then we can ask dc-ops to do the rest.

In T410989#11404665, @cmooney wrote:

@fgiunchedi have all the cables been removed on site?

Typically I would ask DC-Ops to remove in Netbox when they remove on site, to minimize the length of time there is any discrepancy between Netbox and reality.

The logical side on the host side is done. Next up is deleting the interfaces from netbox for the hosts and unplug network cables. I'll file subtasks

I can't currently reproduce the issue -- navigating to https://hub-paws.wmcloud.org/ spawns a container for me and works as expected. Is this still a problem @Sadeiiw67 ?

I'm calling this one done since we have a workaround in place; I'll followup on the Debian bug for an actual fix

In T407140#11383672, @cmooney wrote:

Ok thanks @fgiunchedi for the info.

I think that seems doable. As per the sub-task about a VRF I think that will be needed. And route leaking is not something I really want to do, so we will probably need the "cloudgw" to route traffic between the new VRF and the exsiting cloud-private / Openstack networks.

Reported to Debian as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121006

And lsblk -t for comparison:

As far as I understand the problem, lvm metadata size and alignment can be related to the underlying block device reported data, specifically the "optimal i/o size":

I investigated this a little more today. The issue is grub_lvm_detect allocating memory based on the lvm metadata areas, of which locn[2] is of size 4293914624 (or 0xFFF00000)

I'm giving debugging this issue one more go, as part of this we now have pause-reboot.cfg included for cloudcontrol2010-dev which will avoid mucking on apt1002

Something else I forgot: I'm assuming codfw also is applicable in this case? i.e. these hosts we'll be moving to single nic as well

Thank you @cmooney ! FYI as per Andrew we really only care about cloudcephosd1035 through cloudcephosd1052 since the rest will be decom'd soon anyways

Thank you for following up @Dwisehaupt and I'm glad to know you are making progress! These days I am no longer part of Observability, and will defer to @hnowlan to take it from here

Yes please @cmooney, much appreciated! Note that this is currently not a blocker / not high priority. In the sense that I'll be test-driving the change on 1048 and 1049 which are configured correctly AFAICT, having it done and dusted of course would be great

In T399180#11310972, @cmooney wrote:

In T399180#11310845, @fgiunchedi wrote:

@taavi @Andrew @cmooney what do you think of the above?

The plan sounds good. We need to audit and make sure all the primary links to the cloudcephosd* hosts are set to trunk mode, with the storage vlan as one of the tagged interfaces (in Netbox, and then homer run if we change anything). This change can be made non-disruptively in advance however, so shouldn't be a blocker.

Thank you @cmooney for the summary, I'll add a few thoughts I had while working on the Toolforge on Metal project design document.

Mentioning it here as a followup to T409244: Toolforge outage: toolsdb out of space: it is important we do monitor the ability to read/write toolsdb, and possibly page on it

In T408543#11328875, @cmooney wrote:

1. Move cloudvirt/cloudnet to support jumbo frames ensuring VMs can have a 1500 byte MTU

Doing a comparison with the replica on tools-db-6, there's ~800G free there:

disk space free trend for tools-db-4 over the last 30d

tools-db-4 storage volume is out of space, I'll use this task for tracking

In T408543#11323453, @taavi wrote:

A major downside of doing it via Puppet is that that won't have any effect on instances with non-Puppetized services (so basically anything not managed by WMF SREs).

I'm +1 on getting docker's puppetization to do the right thing, in other words detect the default route's mtu and set said value as docker's default.

In T370037#11299341, @Andrew wrote:

I'm thinking this over again, and I'm fairly compelled by the fact that resources managed by tofu are in source control with a history.

That said, I know that @fgiunchedi found the current workflow very confusing. Filippo, can you talk about your experience a bit here? One thing I wonder about is how both deployments (codfw1dev and eqiad1) are currently coupled, so if tofu can't apply in codfw1dev then we also can't change things in eqiad1; I'm pretty sure that needs to be changed.

Alert is gone, optimistically resolving