In T418899#11840735, @elukey wrote:

[...]

>>> a.components.keys()
dict_keys(['BIOS.Setup.1-1', 'EventFilters.Audit.1', 'EventFilters.Configuration.1', 'EventFilters.Storage.1', 'EventFilters.SystemHealth.1', 'EventFilters.Updates.1', 'EventFilters.WorkNotes.1', 'LifecycleController.Embedded.1', 'RAID.SL.1-1', 'SupportAssist.Embedded.1', 'System.Embedded.1', 'iDRAC.Embedded.1'])

That should be T392851#10975444, so idrac10 hosts, so _disable_lldp in provision may not be compatible with idrac10 yet.

@Jclark-ctr the way it's coded makes it quite flexible, can you try it or ping me with a new server's info so we try it together ? I'll update the doc once we have something fully working.

Yeah, Postgres is where all the data are. So +1 to not backup anything on the frontends.

We're not doing Netbox CSV dumps anymore. So you can remove that directory from backups.

Regarding the count of "prefixes received by the switch but not accepted", I think as a first step it could be a global Netops alert (for our internal peers).
However if it gets triggered more often by server side issues, we should move it to the same "remote_instance" mechanism.

It's great to see progress on this ! Unless there are other blockers I'm unaware off, CDN seems better than LVS as it doesn't require to use a new public IP.

Upgraded to 23.4R2-S8 and all is well.

All done.

I gave a quick review on the CR, but you should at least use https://wikitech.wikimedia.org/wiki/Network_telemetry#remote_instance:gnmi_bgp_neighbor_session_state%7B%7D

@jcrespo
We don't reimage backups hosts. Let us know the alternative method (we can put them out of service for an extended time if necessary). This was brought up in advance @ the IF-DP meeting at the offsite.

Which rack, from each "pods" (see task description) could we use to have an additional "public" vlan ? That means those racks will need to have some space for at least 6 hosts day 1, and probably a few more as time goes. But deployment can be staggered (eg. A/B month X, C/D month Y, etc)

That could help slightly for the few hosts that are in the matching row (allow to move them to a rack without re-numbering).
If we look at eqiad and its 18 hosts, that means ~4.5 per row (which we can round up to 4 as some hosts are to be decom), that means it could make the transition smoother for 6 hosts 3 in A/B, 3 in C/D (assuming the rack we pick already have one public hosts).
To be considered, but not strictly needed.

What would be a good day to alert about those ? Or even better, not even need an alert ?

My initial thought was to start with E/F only but you're right better plan it fully here, especially the IP allocations.

Now that we did the switchover, we could focus more on that upgrade. @Papaul let me know if you're ok to take care of it.

Scheduling this for April 7th at 12:00 UTC - 2h
Pinging @ssingh (Traffic) for visibility.

In T421714#11774619, @BTullis wrote:

I can also take the druid[1012-1013].eqiad.wmnet hosts.

I think that @MoritzMuehlenhoff will likely take krb1002.eqiad.wmnet

For the RIPE atlas we will need to decom it and provision a new one on the future sandbox vlan as IPs will change.
The good news is that the standard IP ranges we use for routed Ganeti are all free (no need to shuffle stuff around like we did in ulsfo).

No deadline, no rush, best effort :)

Ultimately Juniper, I'll take the task for now.

Overall that LGTM, you need to add BGP to security_zones -> production -> services: ['ssh', 'ping', 'traceroute', 'snmp', 'ospf', 'ospf3', 'bgp']

This looks like a monitoring bug, the interface is properly named on the switch. Other metrics are not being collected properly as well. Most likely not exposed properly by the switch.

As a side note we will need to manually change the IPs of the routed ganeti nodes in rack 23 to the 10.128.1.0/24 subnet. Normal operation would have required a re-image but to not lose the VMs and make the migration faster it's best to re-IP the hosts.

All done here. I've also opened T421044: ulsfo: balance VMs between all Ganeti nodes to balance the VMs better.