In T362985#9735205, @ayounsi wrote:

Another question I think is "do we still have to go through text files ?"
It made sens for back in the time when we were manually editing the configuration, and for the few places we still do, but it seems sub-optimal to go from Netbox database to text file to gdnsd.
Probably too simplistic, but could we generate a raw list of IP/FQDN from Netbox, and feed it to gDNSd without having to care about PTR and zones structures ?

In T362421#9710346, @ayounsi wrote:

Prefixes assigned in Netbox: https://netbox.wikimedia.org/ipam/prefixes/?site_id=11

@ssingh I've reserved the following addresses in Netbox for the LVS now, let me know if you need any more info or if I can help.

lvs7001:
  pri_int:
    vlan 711 - public1-b3-magru -  195.200.68.2/27

In T362421#9725493, @jcrespo wrote:

Hi, after 73470d0dca68abee0 ntp no longer auto-restarts, but after one of the latest changes (I believe b48874a81565b7051be39659c056), it is pending. Can it be restarted or should it be kept with the old config for a while, and it should be acked?

In T350179#9725288, @Fabfur wrote:

I'd like to join the chorus of thanks to Papaul, you resolved us a very nasty and long running issue here! Thanks again!

This one in particular down for almost a year and IPs are not responding to ARP/ND on the LAN. Peeringdb still lists them but I removed the session for now, if they get in touch we can set it up again.

I'll take a look and clear up what I can.

Perhaps one option would be to ignore the puppet patch to change drmrs and esams for now - but merge the Homer one and configure magru LVS in puppet to peer with both switches?

I believe the two patches above, once merged, will add the required redundancy. Following option 1 above, creating backup peerings from the LVS hosts to the switch in the other rack.

In T362522#9717511, @cmooney wrote:

FWIW I changed the key-exchange algo configured on mr1-eqsin to see if it would make any difference

In terms of live migration and session re-establishment we should bear in mind that BIRD bgp sessions will use default BGP timers of 180/60 when talking to each other (instead of the 90/30 Juniper defaults).

In T347411#9203210, @cmooney wrote:

Some things that may be possible, if still trying to predict the names from redfish data:

1. Change the NamePolicy to 'mac', so we get names like enxc45ab1abd7f5. Ugly, but should be consistent?

FWIW I changed the key-exchange algo configured on mr1-eqsin to see if it would make any difference, from some brief searching the ec21159 one seems to use less cpu than dh group-exchange one we have now. Thus far hasn't made much impact but I'll recheck the graphs in a day or two.

cmooney@mr1-eqsin# show | compare 
[edit system services ssh]
-    key-exchange group-exchange-sha2;
+    key-exchange curve25519-sha256;

In T362522#9715615, @ayounsi wrote:

it looks like the issue we are facing with the slowness on the device and the reboots is product of a brute force SSH attack on the SRX

The ideal path would require upstream changes in Bird (especially for v6) - http://trubka.network.cz/pipermail/bird-users/2024-April/017580.html

In general I'm a fan of dynamic neighbors so happy to use it here on the Ganeti side.

In T360772#9657554, @ayounsi wrote:

We can define per host hiera keys, and empty lists as well, so to be tested but I don't think we need to implement a new feature

@Papaul yeah I think if we want to go this route we can just set them up the same as we do the netgear or fs.com msw's.

Apologies I'd missed it at first - but there are errors ingress on our circuit from codfw to eqiad on the eqiad side:

In T350179#9692885, @ssingh wrote:

All cp hosts in eqiad are in rows A, B, C, and D, so that does look worth trying out I guess! Can you remind when and if this transition was made in the recent past?

In T361871#9691816, @ayounsi wrote:

We first need to discuss if we want to start using managed switches for management switches (except the aggregation ones).
On the plus side it's convenient to have the extra visibility, but it adds a lots of management overhead to our automation, while I'm not sure we have the resources for that.

In T350179#9586432, @ayounsi wrote:

Last maybe we could explore relying less on PXE, for example is it possible to pass the host and tftp server IP through redfish ?

In T350179#9690121, @ssingh wrote:

Any other opinions/thoughts on how we can try and fix this and where? I am very happy to do the legwork but kind of lost here on what to check next.

This rule in filter table / input chain is allowing the traffic to Hound, but only from two spectific IPs:

6    15097  906K ACCEPT     tcp  --  *      *       172.16.5.238         0.0.0.0/0            tcp dpt:3002
7        0     0 ACCEPT     tcp  --  *      *       172.16.5.200         0.0.0.0/0            tcp dpt:3002

@ayounsi thanks for the patch! LGTM.

I added the above script so we can move this outside of the current provision script, and simplify the work towards replacing that one with one which picks attributes based on the host name/profile.

In T360029#9653766, @Ladsgroup wrote:

In T360029#9635703, @Ladsgroup wrote:

It might sound revolutionary but I think mediawiki should not re-implement DNS. All of these IPs should be removed from dbctl and be done via a fast lookup in mw.

Closing this one, I've made some notes on wikitech below about how to approach these for future rows.

In T345803#9479281, @Papaul wrote:

@cmooney can we get those 2 hosts back in decom? Thanks

Closing this task, everything now completed. For future rows we can base the plan on the steps outlined here:

I've decommed the link from asw-b-codfw to the two Spines now and removed from Netbox. DC-Ops you can proceed and remove the cabling any time, for reference:

In T326322#9130092, @ayounsi wrote:

@cmooney I came across https://www.juniper.net/documentation/us/en/software/junos/interfaces-telemetry/topics/ref/statement/class-of-service-schedulers-queue-monitoring.html not sure exactly what it does though but seems relevant :)

This should be ok from here on. Thanks for the task and troubleshooting!

In T358244#9636601, @ayounsi wrote:

FYI it's alerting for one of its PSU being down, but we don't really care anymore :

asw-a-codfw> show system alarms
1 alarms currently active
Alarm time Class Description
2024-03-16 09:20:23 UTC Major FPC 6 PEM 1 is not powered

I downtimed the stack in Icinga/LibreNMS for 1 month

To confirm all the new networks in codfw are within 10.192.0.0/16, which I believe should be ok in terms of grants.

Thanks for confirming @KCVelaga_WMF. I’ll get that done over the next day or so; we have our annual SRE meet up this week but I should get a few minutes to make the changes.

Thanks for looking at this @ayounsi. It was an oversight I'd not fully tested but yes everything was mostly ok apart from the log message trying to access .name. Patch should fix.

In T326322#9615636, @fgiunchedi wrote:

Yeah having some ballpark numbers will be a great help @cmooney, unless we're talking hundreds of thousands more metrics than we have now I think we're good to go, tens of thousands we can do without much effort/resources

@fgiunchedi wondering if you'd any thoughts on the above suggestion to allow more series through from the gnmic pipeline?

This is likely what also happened with T358099, I'll have a look and see if I can work out where it's going wrong.

Error due to define virt_floating = ["185.15.56.0/25"]:

Mar 08 13:10:57 cloudgw1002 nft[42895]: In file included from /etc/nftables/main.nft:9:1-36:
Mar 08 13:10:57 cloudgw1002 nft[42895]: /etc/nftables/110_cloudgw_puppet.nft:184:43-55: Error: unknown identifier 'virt_floating'
Mar 08 13:10:57 cloudgw1002 nft[42895]:         ip saddr { $virtual_subnet_cidr, $virt_floating } accept
Mar 08 13:10:57 cloudgw1002 nft[42895]:                                           ^^^^^^^^^^^^^

@ayounsi finally got back to this for a closer look. Really great work, I tried to make a device-centric dashboard here also.

In T358658#9605712, @KCVelaga_WMF wrote:

@cmooney I have moved over the files from stat1005:kcv-wikimf to stat1008:kcvelaga, and everything is working fine.

@KCVelaga_WMF great news!

Closing task. Big thanks to all the SRE teams for the help and co-operation getting this one over the line :)