In T440999#12426889, @RobH wrote:

I understand this a bit better, it would be mainly for the 85 wikikube hosts in rows A/B

Thanks @RobH, fwiw I did a monitor on the port too and there are no frames incoming to us. Worth checking that the IPs aren't changing (it's ok if they are) and that everything is properly configured on the provider side (probably they need to do something still).

In T440696#12420494, @MLechvien-WMF wrote:

Hi John,

Yes that approach sounds fine, thanks.

I've made a start on this tackling ssw1-d1-eqiad first, and will do the same on ssw1-d8-eqiad in a few days if all seems ok.

In T440631#12424104, @ayounsi wrote:

We should probably try to cross check devices:

• Reboot the mgmt switch connected to the mgmt port showing the issue
• Reboot the nokia switch connected to the neighbor switch

@ssingh @Vgutierrez please advise if this is the correct approach to be taking here. I know we have made a lot of progress with IPIP, but I'm not sure if we are at a point we can instead just get rid of all the vlans on lvs1019? Or if instead we should follow this plan and add the new row A/B vlans to it (and the backup).

Closing this now, the patch has been merged to support this in our automation, and some validation checks to make sure the interfaces are sane.

Ok I think this should be sorted now. Root cause was that we were relying on the in-built trust anchors powerdns-recursor ships with, rather than those in /usr/share/dns/root.key with are managed by the dns-root-data package. So we had not loaded the new 38696 KSK. When the root zone began to be signed with that new key starting yesterday we were unable to validate the signatures, resulting in the SERVFAILs.

In T440830#12420954, @dkertesz wrote:

I'm not familiar with these systems so take this with a big grain of salt: it seems that pdns only knows about the old key (rec_control get-tas) and is not loading /etc/powerdns/recursor.lua, which eventually loads /usr/share/dns/root.key which contains both the old and the new key.

doh3005 seems ok fwiw

cathal@officepc:~$ dig +nsid +dnssec +https www.ietf.org @wikimedia-dns.org

In T436181#12417516, @Jclark-ctr wrote:

@cmooney I have these three servers I'd like to rack in A3, B3, and C1 using the new Nokia switches. If you have any concerns, let me know. Otherwise, I'll plan to rack them next week, if that works for you.

Cool thanks. If we can run the following links from the Juniper switches in those racks to the new ones that'd be great. No rush at all though it'll be a few weeks before I can look at it. We can use the 40GBase-SR4 optics that had been connecting the old C/D Junipers we were talking about (and the pink MPO fibres).

In T418012#12411083, @Jclark-ctr wrote:

Finished cabling for the A/B upgrade. Everything is complete except for 1 cross-connect between the cages Cables are pre ran though.

In T440068#12416633, @Dzahn wrote:

Just checking on this part. This should be about "esams <-> eqiad" link. the currently active backend is releases1003 in eqiad.

@brouberol we were delayed waiting on dc-ops to get the cabling done but we’ve made progress on that this week so things are finally shaping up.

@Jclark-ctr moved the uplink to the adjacent rack and everything is now working fine.

In T440068#12414454, @Dzahn wrote:

• roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo transport, or the cp side

The issue is the speed at which esams is pulling the file in from codfw at, which happens every time. Tbh it doesn't make much sense for users to connect to esams for this given that is the case, but that is how the CDN currently works.

cmooney@bast3007:~$ wget https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz 
--2026-10-09 00:03:33--  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz
Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93194737 (89M) [application/x-gzip]
Saving to: ‘mediawiki-1.43.9.tar.gz.1’

I've upgraded cr1-codfw now too so we can close this.

In T418018#12403321, @Jclark-ctr wrote:

@cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS port labels and Google Sheet. Adjusted the baud rate and confirmed everything worked.

@Urbanecm_WMF I've added those vars to hieradata/cloud/eqiad1.yaml now which I think is the best place for them to be (along with the definitions which are causing the class to be loaded). So I think you can probably remove them from the cloud/instance-puppet repo where you added them, although likely they'll not cause any problems if they stay there too.

In T420223#12403094, @elukey wrote:

The first column is Kbps in, so RX for eno1, that is the main host interface for wikikube-worker1156. That value indicates a saturation of the 1G NIC when receiving data.. So this is probably the cause of the jitter, and mcrouter is so sensitive that is affected.

In T417051#12401539, @cmooney wrote:

Fundamentally I would suggest the issue here is the use of static routes that did not respond to the link failure, and use a different path. I think reviewing the network design here is the better way to approach this, removing those static routes so it cannot happen again.

I'm not sure we want to create compound alerting rules like that.

In T440201#12396248, @Urbanecm_WMF wrote:

FWIW I have no idea whether adding those values where I added them is a good long term fix. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040158 shows they possibly should be in puppet somewhere.

In T440201#12396219, @Urbanecm_WMF wrote:

I added missing values to deployment-prep: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e03f2606f899152f20a6401452175f59a5619e10%5E%21/#F0

Now puppet runs.

In T438876#12396105, @Jhancock.wm wrote:

@cmooney the console port on the atlas on port 47, there is no cable connected to it and it doesn't seem to have an approriate console port. not without using a 9 pin. i can fix this if we want to do that. or disable the port.

I'm going to close this one for now as we know the cause.

In T417871#12382617, @Papaul wrote:

I spoke with @ssingh and @BCornwall to see if we can schedule the core routers Junos upgrade next week Wednesday at 9am CT, @ssingh will get back in touche with me on the confirmation.

In T439560#12378680, @RThomas-WMF wrote:

2. Do you know the timeline to address this open request: T429175?

Ok I see what happened here. The patch I linked in the above message was merged on the 21st, however due to how the network names are mapped it wasn't added to the Squid ACL until a second patch was made on Sept 25th.

With fresh eyes the logs are there I was filtering incorrectly. I can see the connection at 12:19 on Sept 24th was rejected with a 403 code by the proxy:

{
  "http.response.bytes": 3533,
  "event.category": [
    "network",
    "web"
  ],
  "labels.hierarchy_status": "HIER_NONE",
  "type": "syslog",
  "facility": "local0",
  "program": "squid",
  "rsyslog.hostname": "urldownloader1004",
  "rsyslog.facility": "local0",
  "host.domain": "wikimedia.org",
  "host.hostname": "urldownloader1004.wikimedia.org",
  "http.request.method": "CONNECT",
  "event.outcome": "unknown",
  "url.domain": "rolesanywhere.us-east-1.amazonaws.com",
  "rsyslog.timereported": "2026-09-24T12:19:40.727649+00:00",
  "url.path": "-",
  "source.ip": "10.68.118.192",
  "@timestamp": "2026-09-24T12:19:40.783Z",
  "host.name": "urldownloader1004",
  "ecs.version": "1.7.0",
  "@version": "1",
  "http.request.referrer": "-",
  "event.type": [
    "access",
    "connection"
  ],
  "severity": "info",
  "event.dataset": "squid.access",
  "rsyslog.severity": "info",
  "rsyslog.programname": "squid",
  "host.ip": "208.80.155.118",
  "service.type": "squid",
  "tags": [
    "input-kafka-rsyslog-shipper",
    "rsyslog-shipper",
    "kafka",
    "es"
  ],
  "logsource": "urldownloader1004",
  "http.version": "1.1",
  "event.duration": "-",
  "url.full": "rolesanywhere.us-east-1.amazonaws.com:443",
  "host": "urldownloader1004",
  "client.ip": "10.68.118.192",
  "event.kind": "event",
  "labels.request_status": "TCP_DENIED",
  "timestamp": "2026-09-24T12:19:40+0000",
  "http.response.status_code": 403,
  "user_agent.original": "Go-http-client/1.1"
}

In T439560#12377843, @RThomas-WMF wrote:

Hence, this suggests the request may be rejected by the proxy --> urldownloader before reaching AWS. If AWS rejects a call, boto3 will log the return code 4xx or 5xx depending on the error and not a StatusCode: 0

Thanks for the task @KMontalva-WMF.

No objection to us working with Juniper to fix this, or even the workround with the description / rewrite.

Thanks @CDobbins for getting to the bottom of this!

@MoritzMuehlenhoff to get some clarity I checked that ranges the patch is adding by swapping out mw_appserver_networks for production_networks.

@Jclark-ctr @VRiley-WMF, going through the ports on scs-a8-eqiad these were the issues I found.

In T438161#12350289, @Jclark-ctr wrote:

This is cabled with 10g. @cmooney i missed that also. I would think it should work being setup as 10g.

In T430658#12345413, @BTullis wrote:

So I think I will need to update the DNS repo for this, as all of the other pod zones appear here templates/10.in-addr.arpa.
This is related to work being undertaken on: T376291: Authdns: automate reverse DNS zone delegation for k8s pod IP ranges

Yeah it's not a network level block:

cmooney@an-conf1004:~$ mtr -b -w -c 10 --tcp -P 8080 urldownloader.svc.eqiad.wmnet 
Start: 2026-09-22T11:42:03+0000
HOST: an-conf1004                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae1-1030.cr1-eqiad.wikimedia.org (10.64.5.2)  0.0%    10    0.5   1.1   0.4   6.3   1.8
  2.|-- urldownloader.svc.eqiad.wmnet (10.2.2.95)     0.0%    10    0.4   0.4   0.4   0.4   0.0

Squid is rejecting it though:

cmooney@an-conf1004:~$ curl --proxy http://url-downloader.wikimedia.org:8080 https://www.google.ie
curl: (56) CONNECT tunnel failed, response 403

In T437560#12312244, @MoritzMuehlenhoff wrote:

As for

As seen in T437439, when the urldownloader proxy were put behind LVS, access from the analytics VLAN was dropped

I don't believe this to be true: Looking at the git history of profile::url_downloader access was already restricted to the networks used by the mediawiki app servers (and later wikikube). All other use cases simply always only used the web proxy caches.