Agh hit a bit of a hiccup with this (really should have anticipated). Take drmrs for example:

cmooney@asw1-b12-drmrs> show configuration interfaces et-0/0/48 
description "Core: cr1-drmrs:et-0/0/1 {#D0100}";
mtu 9192;
unit 0 {
    family inet {
        address 185.15.58.143/31;
    }
    family inet6 {
        address 2a02:ec80:600:fe06::2/64;
    }
}

In T424683#11906733, @cmooney wrote:

In T424683#11885878, @ayounsi wrote:

Nice!

We can also filter out the .16386, .16384, .16385, .16383, .32769 - weird juniper...
as well as .0 like we do for SNMP (for LibreNMS) - see T283060: SNMP: filter out default sub interfaces

We don’t get the .0 via gnmi so one less worry.

But yep, sounds like it makes sense to drop the others if needed. I’ll try to validate what ones we get back in stats.

So anyway, for now I'd propose we add the following vlans for this:

341  core1-bw27-esams
342  core1-by27-esams

All looks good with the link, traffic flowing again.

It still shows no light incoming to asw1-b12-drmrs on lane 3:

cmooney@asw1-b12-drmrs> show interfaces diagnostics optics xe-0/0/50:2 | match "Laser receiver power" | match dB    
    Laser receiver power                      :  0.000 mW / -40.04 dBm

In T424611#11886556, @ayounsi wrote:

I suggest core1 instead of corebgp but that lgtm!

Patch merged and config pushed to all Nokia devices now.

@RobH can you raise a task with Digital Realty to take a look at this in MRS2?

Very weird, box still sees the optic but thinks interface is invalid.

cmooney@asw1-b12-drmrs> show chassis pic fpc-slot 0 pic-slot 0 | match "^  50" 
  50   40GBASE SR4       MM    FS                 QSFP-SR4-40G      850 nm                    0.0           REV 01   SFF-8436 ver n/a

cmooney@asw1-b12-drmrs> show interfaces et-0/0/50 detail                          
error: device et-0/0/50 not found

In T424683#11885878, @ayounsi wrote:

Nice!

We can also filter out the .16386, .16384, .16385, .16383, .32769 - weird juniper...
as well as .0 like we do for SNMP (for LibreNMS) - see T283060: SNMP: filter out default sub interfaces

In T425674#11899170, @taavi wrote:

It seems like the same problem is present on all VLAN interfaces on cloudvirt/cloudcontrol hosts:

taavi@cloudvirt1076 ~ $ ip link show vlan1105
13: vlan1105@eno12399np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 04:32:01:db:42:d0 brd ff:ff:ff:ff:ff:ff

So it seems the reason for this is some ferm complexity. When puppet signals a 'refresh' to it it asks ferm itself to reload rules, however ferm does not pick up on the changes to these files for some reason. The ultimate answer is for hosts to move to nftables, it's not worth spending time to fix.

Just an update here.

10.2.2.30 is the IP for search.svc.eqiad.wmnet, so it looks like cloudelastic1012 is set up for that rather than cloudelastic.wikimedia.org.

In T422816#11878805, @elukey wrote:

It doesn't seem a propagation error related to ferm, but it should be something related to it if I had to guess. No errors reported after it, so we should be good.

In T422816#11876405, @herron wrote:

@ayounsi @cmooney would you be able to have a look to see if anything stands out from netops perspective?

@Papaul thanks. I see most of those don't exist even for IPv4, nor are there any IPv6 addresses listed, so I'm not sure exactly what might need to be added.

I had a stab at this in the above patch. Some notes on the event processors added:

My basic thoughts on this are:

In T390052#11859386, @ayounsi wrote:

Now we need to figure out if it's worth upgrading the management routers or not, as it's more recent that the current Junos Recommended Version.

IPv4 looks ok:

Target: 208.80.154.224
Location: USA, CA, Fremont
Provider: Linode
MTR captured UTC: 2026-04-17T17:02:48.834Z
MTR captured local: Fri Apr 17 2026 18:02:48 GMT+0100 (Irish Standard Time)

3 working example traces with mitigation off:

Target: 2620:0:861:ed1a::1
Location: USA, TX, Houston
Provider: Cubepath
MTR captured UTC: 2026-04-17T16:47:50.831Z
MTR captured local: Fri Apr 17 2026 17:47:50 GMT+0100 (Irish Standard Time)

Re-opening this so we can look at the automation changes that are needed to remove tthe OSPF stuff (see here).

In T356877#11814024, @JMeybohm wrote:

We can also run the prometheus bird exporter as a sidecar to calico-node (container image needs to be build and some yaml needs to be edited) which should give us more standardized metrics. We could add that as a stretch goal for this task.

Broadly the patch submitted looked good to me, though I see it was abandoned.

In T422043#11804537, @Jhancock.wm wrote:

D row has no specialty rack at all so we can easily work around that for future private vlan installs.

Ok Juniper came back with the following:

I found that your version 23.4R2-S7.4 is hitting the PR1933049. Unfortunately, this is a confidential PR, but in order to get this issue resolved and avoid further issues, you need to upgrade to a slightly higher version.

Ok the results from wikikube-worker1258 (row B) don't seem to show the same percentage of longer RTT packets as wikikube-worker1273 (row D - in above comment).

Bucket            Count     Pct  Bar
--------------------------------------------------
> 500ms               1  0.001%  
250 - 500ms           0  0.000%  
100 - 250ms           0  0.000%  
50 - 100ms            0  0.000%  
40 - 50ms             2  0.001%  
30 - 40ms        136548  99.998%  █████████████████████████████████████████████████
20 - 30ms             0  0.000%  
10 - 20ms             0  0.000%  
5 - 10ms              0  0.000%  
2 - 5ms               0  0.000%  
1 - 2ms               0  0.000%  
0 - 1ms               0  0.000%

Is it maybe an idea to re-use some of the existing vlans? Like if we assign rack A1 as the public rack for the A/B POD we could add all the hosts to public1-a-eqiad as we move them? And then when complete rename the vlan to public1-a1-eqiad?

We are hopeful the situation should have improved after codfw was repooled, adding additional capacity. Root cause of the circuit breaking is still being investigated.

In T420223#11781147, @elukey wrote:

@cmooney I added some info T420223#11753137, where I tested jitter seen by MTR on a worker in row A/B vs a worker in C/D: the former doesn't show it. I also tried on another couple of nodes, but I don't have anything definitive form a statistics point of view. I can collect more info if you want!

If we are going to have one public-enabled rack per "pod" then should we not have just one vlan assigned for codfw row E/F (and then one also for a/b and c/d)?

In T420223#11776822, @cmooney wrote:

If there is a wikikube-worker in rows a/d with mcrouter regularly talking to codfw mc hosts let me know, I can potentially do the same kind of analysis on traffic there so we can compare the difference?

Ok so I gathered stats for the past few days (Mar 27 - Apr 1) of the SYN / SYN-ACK exchanges starting the tcp handshake, and this is the breakdown of RTTs:

Total SYN / SYN-ACK RTTs measured: 146553

In T420223#11758444, @elukey wrote:

@cmooney yes Effie depooled it IIRC! You can probably use wikikube-worker1273.eqiad.wmnet (@jijiki let's not depool it).

In T416249#11758413, @Jclark-ctr wrote:

@cmooney Could you assist with this next week?

In T420223#11753682, @cmooney wrote:

I'll do another pcap and just focus on SYN / SYN-ACK packets, which will be more reflective on the network latency

Thanks for the write-up @JMeybohm. Definitely an odd one.

I took a pcap on wikikube-worker1070 for TCP packets to mc1041, and did some comparisons on RTT (i.e. time between packet sent to mc1041 and the response arriving.

Total RTT samples: 280984

In T420223#11752904, @elukey wrote:

Much better. @cmooney nothing definitive because there may be some variance but what do you think?

This should now be working again. Big thanks to @ayounsi for the heavy-lifting with all the puppet patches to add the $INSTALL_HOSTS set.