@Jhancock.wm @Papaul I'd been using the server in b7 for testing already, but I should be able to move over to the one in a8 instead (I assume we have the same problem with public1-a-codfw as we had with public1-b-codfw)

So some interesting findings when testing today.

In T362421#9808627, @ayounsi wrote:

The Telxius community doesn't seem to be of any effect so far, I'll wait for their reply, maybe they changed or need to be enabled on their side first. I'll look at the other providers afterwards.

From what I can tell the 'authoritative' statement only controls NAK generation. I think we're hitting this part of the code, and the different source address (of another switch) on the duplicate REQUESTS is why it is sending the NAKs:

Re-reading the man page for dhcpd.conf it seems that pontentially changing the 'authoritative' statement at the top of our config to 'not authoritative' would prevent it sending the NAKs. Might be worth a shot? Better to not create them than to filter them elsewhere. I don't believe in our environment there is any use-case where we need NAKs.

Just a note on this, I only discovered this document after the task:

In T362421#9808194, @ayounsi wrote:

They might prefer going through EdgeUno once we add the prepending to Novvacore, so the same change would be needed there as well.

Also I didn't see in the dhcpd docs and way to constrain the generation of NAKs in response to invalid REQUEST messages.

One observation is that the NAK's are unique in so far as they are sent from 208.80.153.33 (Switch IRB int IP) to 255.255.255.255 (and matching L2 MACs).

Pcap of DHCP request from contint2002 here:

In T359054#9807307, @CDanis wrote:

Adding the 3rd transit link in magru greatly improved the latency for many users in Argentina.

+1 sounds like a good idea. Nice we have some limited scope to experiment with the DoH ranges before pulling the plug on ns2.

This has been implemented and the new vlan setup is recorded here. Closing task

Thanks. It is very much something we wish to do but unfortunately other priorities have always trumped it for multiple past quarters.

And fwiw announcement looks good, all 3 of our transits are learning it ok, and I see it on other carriers from those sources as well. We also see live requests on the doh servers.

In T364893#9800673, @BTullis wrote:

I think that the most likely candidate at the moment is user-generated load from someone doing a heavy query in Superset, thereby pulling lots of data from HDFS, via the presto workers.

Gonna close this one. As a last datapoint if you 'stack' the Hadoop graph in Grafana you can clearly see the cumulative reads at ~15:55 on May 14th was a good deal higher than any of the other spikes of usage over the past few days (peaks at almost 200Gbit/sec). So it makes sense that paged and the others didn't.

In T364893#9799598, @JAllemandou wrote:

The amount of data read varies a lot depending on jobs. One can see that the amount of data read from HDFS is very spiky: https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&var-hadoop_cluster=analytics-hadoop&from=now-7d&to=now&viewPanel=111

Seems this is not possible as the cloudsw's still on JunOS 18 don't support exporting the data within the mgmt routing-instance.

Thanks for the task and analysis.

Patch to Homer wmf plugin merged now, so BGP to VMs at POPs / on L3 switches now under automation too.

In T363702#9794749, @Volans wrote:

I think that the proposed check covers only a very specific failure scenario that is unlikely to happen again before Liberica

Thanks for the task @taavi. Looks well put together let me know the exact time you're starting and if feel free to ping me if there is anything you need checked from the physical network side of things (where MAC addresses are in the forwarding tables etc.)

In T187929#9793592, @ayounsi wrote:

@cmooney what do you think of duplicating the other POPs allocation scheme?
For example looking at eqiad as example, keep 2a02:ec80:a000::/40 as "reserved for future growth"
Then use 2a02:ec80:a000::/48 for the existing WMCS eqiad infra
Then 2a02:ec80:a000::/56 for public, another /56 for private, /55 for the infra, 2a02:ec80:a000:ed1a::/64 for VIPs, etc

Happy to discuss. I think if we are doing this it makes sense to do the cloudgw <-> cloudsw BGP at the same time (we will need to create the Bird config for the cloudgw to talk to cloudnet, so while we are doing so let's do the other side too).

In T364559#9784420, @Andrew wrote:

Reimaging cloudcontrol2006-dev works now, thanks!

@Papaul I've added all the links for the new switches in Netbox now:

@ayounsi @cmooney please provide the interfaces to use on cr* for the uplink from ssw1-d1 to cr1 and ssw1-d8 to cr2

Hey Andrew,

In T354872#9529469, @MatthewVernon wrote:

Sorry, I think object stores are often not really written with renumbering in mind...

Sorry for the delay, the capirca script times out a lot for some reason will need to look at that.

In T364454#9780995, @ssingh wrote:

On mr1-magru, I see 10.140.1.18 (prometheus7001) and denied by policy, which makes me wonder if we need to run https://netbox.wikimedia.org/extras/scripts/capirca.GetHosts/ for Capirca to generate the ACL?

In T364464#9780633, @Papaul wrote:

@cmooney I think this is just a human error issue. We were racking all the lsw1-d* yesterday and maybe we accidentally bumped into the cable. We will check once on site.

Thanks

@Papaul I think this one is ready to be moved to rack D1 now.

Device has been removed from LiberNMS now. I also downtimed it for 2 weeks just in case I mess up the order of anything.

In T345823#9766574, @JMeybohm wrote:

50 should be absolutely fine. My theory is that while I was moving around Pods and IP blocks in the cluster(s), calico tried to be clever (keep connectivity as far as possible) and started announcing /32 "blocks" for a bunch of pods - which tripped the limit.

In T345823#9763845, @JMeybohm wrote:

We decided during migration of production to a bigger Pod IP space that this will not be necessary for staging and it actually is not. The issue there (as we figured later) that the IP space is split into /26 blocks, effectively limiting the cluster size to 4 nodes (including control-plane). The change to the IP block size was made to overcome this limitation without the need of changing the Pod IP space (and therefore having to reconfigure that in various places).

Not sure if it might be worth taking a step back and weighing up what's happening here?

These are direct peerings to Equinix tehmselves over their own exchange. We are waiting on them to complete the configuration of their side (see peering@wikimedia.org). I emailed last week to chase them for an update.

Looks like this was a brief blip of inbound errors (unlike last time when they began and kept increasing until eventually the link failed).

Actually it may be just easier to check the route for each pooled IP and make sure the check doesn't return saying it's using the default as per the task descr.

cmooney@lvs1019:~$ ip --json route get fibmatch 1.1.1.1 
[{"dst":"default","gateway":"10.64.32.1","dev":"eno1np0","flags":["onlink"]}]

Thanks Brian.

In T362746#9747411, @cmooney wrote:

In T362746#9745233, @ABran-WMF wrote:

good catch! at this point I think it's not a concern anymore. Unless it has further implications?

No, the fact it worked means we didn't hit the issue we have seen before with this, so no further action needed.

I note the 10G NIC model in this box is a BCM57412, I think perhaps the issue we seen before only applies to BCM57810, I'll maybe look more into that and discuss with DC-Ops but not relevant to this task. Thanks!