Page MenuHomePhabricator

cmooney (Cathal Mooney)
SRE (netops)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
May 10 2021, 3:25 PM (162 w, 5 d)
Availability
Available
IRC Nick
topranks
LDAP User
Cathal Mooney
MediaWiki User
CMooney (WMF) [ Global Accounts ]

Recent Activity

Fri, Jun 21

cmooney closed T365987: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad, a subtask of T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2, as Resolved.
Fri, Jun 21, 10:41 PM · Infrastructure-Foundations, netops, SRE
cmooney closed T365987: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad as Resolved.
Fri, Jun 21, 10:41 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney added a comment to T367439: No unicast IP ranges announced to peers from eqdfw.

Gonna copy some of the discussion from the patch here as I think it's easier for discussion and a record of what we decide:

Fri, Jun 21, 1:30 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney created P65307 Routes matching updated regex to also include Anycast ASN at eqord.
Fri, Jun 21, 1:26 PM
cmooney created P65305 Routes matching updated regex to also include Anycast ASN at eqdfw.
Fri, Jun 21, 1:21 PM
cmooney created P65304 (An Untitled Masterwork).
Fri, Jun 21, 1:03 PM

Thu, Jun 20

cmooney triaged T368084: Update esams network pop diagrams as Medium priority.
Thu, Jun 20, 4:22 PM · Infrastructure-Foundations, netops, SRE
cmooney added a comment to T365987: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad.

Upgrade complete, all looks good network side.

Thu, Jun 20, 3:18 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney closed T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad , a subtask of T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2, as Resolved.
Thu, Jun 20, 11:41 AM · Infrastructure-Foundations, netops, SRE
cmooney closed T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad as Resolved.
Thu, Jun 20, 11:41 AM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE

Wed, Jun 19

cmooney closed T367408: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades? as Resolved.

Cool thanks @Papaul. I guess we can see how we get on over the next while. If the number of 1G -> 10G upgrades become excessive we might have to review as moving rack also means the SREs who manage the system have to reimage/renumber it (moving rack vlan), so it adds an extra layer of complexity for them. If they are only occasional hopefully that won't become a major pain point.

Wed, Jun 19, 4:33 PM · netops, Infrastructure-Foundations, SRE
cmooney created P65205 (An Untitled Masterwork).
Wed, Jun 19, 12:12 PM
cmooney created P65202 (An Untitled Masterwork).
Wed, Jun 19, 11:44 AM
cmooney added a comment to T367959: Mailman PTR records.

We've run into at least one instance with rsync that needed changing, rsyncd looks up the IP address and since it sometimes got lists1004 and sometimes got lists, was intermittently failing needing this fix.

Wed, Jun 19, 11:32 AM · collaboration-services
cmooney created P65193 (An Untitled Masterwork).
Wed, Jun 19, 8:27 AM

Tue, Jun 18

cmooney added a comment to T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad .

Switch is back online after upgrade, everything looks good at first glance.

Tue, Jun 18, 3:18 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney committed rOSHO96d281339a96: CHANGELOG: add changelogs for release v0.6.6.
CHANGELOG: add changelogs for release v0.6.6
Tue, Jun 18, 3:05 PM
cmooney committed rOSHO7af405b0dd0e: Include vlans with an IRB int in device vlans even if not on L2 port.
Include vlans with an IRB int in device vlans even if not on L2 port
Tue, Jun 18, 2:26 PM
cmooney created P65153 IPv6 dns additions serviceops hosts.
Tue, Jun 18, 11:12 AM

Mon, Jun 17

cmooney renamed T364092: Upgrade core routers to Junos 22.4R3 from Upgrade core routers to Junos 22.2R3 to Upgrade core routers to Junos 22.4R3.
Mon, Jun 17, 4:56 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T364092: Upgrade core routers to Junos 22.4R3.

Both Junos 22.2R3-Sx and Junos 22.4R3 are latest recommended. fyi, I went with 22.4R3 in magru.

Mon, Jun 17, 4:24 PM · netops, Infrastructure-Foundations, SRE
cmooney updated the task description for T364092: Upgrade core routers to Junos 22.4R3.
Mon, Jun 17, 4:24 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T367732: POPs LVS : remove public vlan trunking.

Slight drawback here is that in ulsfo/eqsin LVS traffic towards public hosts (if any) will hair-pin through the routers and back to the switches.

Mon, Jun 17, 3:19 PM · netops, Infrastructure-Foundations, Traffic
cmooney triaged T367573: Update fundraising mail / firewall settings to use new production mx-in hosts as Medium priority.
Mon, Jun 17, 3:17 PM · fundraising-tech-ops, Infrastructure-Foundations, Mail, SRE
cmooney added a comment to T367408: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades?.

@ayounsi perhaps I was a little quick to conclude all the blocks were assigned, you are correct.

Mon, Jun 17, 9:51 AM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T250415: Homer: add parallelization support.

For my part I like “3” as set out by Volans above.

Mon, Jun 17, 9:24 AM · User-Elukey, Infrastructure-Foundations, SRE-tools, homer

Fri, Jun 14

cmooney created P64986 (An Untitled Masterwork).
Fri, Jun 14, 7:19 PM
cmooney updated the task description for T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches.
Fri, Jun 14, 4:58 PM · netops, Infrastructure-Foundations, SRE
cmooney updated subscribers of T367439: No unicast IP ranges announced to peers from eqdfw.
Fri, Jun 14, 1:55 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney added a parent task for T367512: Get test host connected to codfw row c/d lsw's: T364095: Codfw row C/D switch installation & configuration.
Fri, Jun 14, 1:22 PM · DC-Ops, ops-codfw, SRE, netops, Infrastructure-Foundations
cmooney added a subtask for T364095: Codfw row C/D switch installation & configuration: T367512: Get test host connected to codfw row c/d lsw's.
Fri, Jun 14, 1:22 PM · DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE
cmooney triaged T367512: Get test host connected to codfw row c/d lsw's as Medium priority.
Fri, Jun 14, 1:22 PM · DC-Ops, ops-codfw, SRE, netops, Infrastructure-Foundations

Thu, Jun 13

cmooney added a comment to T367439: No unicast IP ranges announced to peers from eqdfw.

Just to note that for the same time period (since March 5th) we've not been announcing the codfw aggregates from eqord:

cmooney@cr2-eqord> show route advertising-protocol bgp 192.80.17.197 terse 208.80.152.0/22
Thu, Jun 13, 9:44 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney created P64895 Routes matched by updated as-path regex in eqdfw aggregate.
Thu, Jun 13, 9:34 PM
cmooney added a comment to T367439: No unicast IP ranges announced to peers from eqdfw.

I'm monitoring the change in traffic levels. Right now it seems negligible, however that is not much surprise, prior to the LVS peering change that caused us to stop sending them fom codfw (T352920) they also barely showed on the graphs. The biggest ingress we see now is over peering at Equinix and it's still only about 20Mbit:

Thu, Jun 13, 9:28 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney added a comment to T367439: No unicast IP ranges announced to peers from eqdfw.

I've pushed this change to cr2-eqdfw and it seems to be doing what we need there:

Thu, Jun 13, 9:18 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney added a comment to T367439: No unicast IP ranges announced to peers from eqdfw.

It seems this was an inadvertent result of the upgrade to the codfw row A/B switches, and the move there from a purely L2 switching layer to a routed one.

Thu, Jun 13, 8:32 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney renamed T367439: No unicast IP ranges announced to peers from eqdfw from No IPv6 ranges announced to peers from eqdfw to No unicast IP ranges announced to peers from eqdfw.
Thu, Jun 13, 4:30 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney triaged T367439: No unicast IP ranges announced to peers from eqdfw as High priority.
Thu, Jun 13, 4:05 PM · Patch-For-Review, Infrastructure-Foundations, netops, SRE
cmooney closed T365983: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad as Resolved.

Thanks for checking things, all stable on our side I will close the task now.

Thu, Jun 13, 3:57 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney closed T365983: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad , a subtask of T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2, as Resolved.
Thu, Jun 13, 3:56 PM · Infrastructure-Foundations, netops, SRE
cmooney added a comment to T365983: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad .

Switch has reloaded on the new version, all looks good at first glance.

Thu, Jun 13, 3:22 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney added a comment to T367408: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades?.

We could use these cables but the host side but we might not have enough slack to connect to servers at different heights in the rack:

Thu, Jun 13, 1:42 PM · netops, Infrastructure-Foundations, SRE
cmooney closed T365982: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad, a subtask of T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2, as Resolved.
Thu, Jun 13, 1:01 PM · Infrastructure-Foundations, netops, SRE
cmooney closed T365982: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad as Resolved.
Thu, Jun 13, 1:01 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney triaged T367408: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades? as Low priority.
Thu, Jun 13, 12:50 PM · netops, Infrastructure-Foundations, SRE

Tue, Jun 11

cmooney lowered the priority of T367199: hw troubleshooting: Faulty 40GBase-LR4 link from cloudsw1-d5-eqiad to cloudsw1-f4-eqiad from Medium to Low.

Thanks for the help with this @VRiley-WMF. The link has now been clean for over an hour so I think we can say for certain the optic from cloudsw1-f4-eqiad et-0/0/54 was faulty. We should dispose of that one now.

Tue, Jun 11, 6:30 PM · netops, Infrastructure-Foundations, SRE, ops-eqiad, DC-Ops
cmooney updated the task description for T367199: hw troubleshooting: Faulty 40GBase-LR4 link from cloudsw1-d5-eqiad to cloudsw1-f4-eqiad.
Tue, Jun 11, 4:41 PM · netops, Infrastructure-Foundations, SRE, ops-eqiad, DC-Ops
cmooney added projects to T367199: hw troubleshooting: Faulty 40GBase-LR4 link from cloudsw1-d5-eqiad to cloudsw1-f4-eqiad: Infrastructure-Foundations, netops.
Tue, Jun 11, 4:39 PM · netops, Infrastructure-Foundations, SRE, ops-eqiad, DC-Ops
cmooney renamed T367199: hw troubleshooting: Faulty 40GBase-LR4 link from cloudsw1-d5-eqiad to cloudsw1-f4-eqiad from hw troubleshooting: bad fiber cable between cloudsw1-d5-eqiad port et-0/0/53 to cloudsw1-f4-eqiad port et-0/0/54 to hw troubleshooting: Faulty 40GBase-LR4 link from cloudsw1-d5-eqiad to cloudsw1-f4-eqiad.
Tue, Jun 11, 4:38 PM · netops, Infrastructure-Foundations, SRE, ops-eqiad, DC-Ops
cmooney triaged T367203: Sub-optimal cloud routing for WMCS in eqiad when link fails as Low priority.
Tue, Jun 11, 4:11 PM · Infrastructure-Foundations, netops, SRE
cmooney added a comment to T365982: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad.

Switch has reloaded on new version and initial checks look ok.

Tue, Jun 11, 3:20 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE

Mon, Jun 10

cmooney closed T325801: Juniper QFX5120 error logs on lsw1-e1 and lsw1-f1: Failed to get ifl for ifl index as Resolved.

We seem to have no such errors being logged any more, either from these switches or the dedicated spines in eiqad. Gonna close.

Mon, Jun 10, 3:13 PM · Infrastructure-Foundations, SRE, netops
cmooney triaged T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad as Medium priority.
Mon, Jun 10, 2:25 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2.
Mon, Jun 10, 12:33 PM · Infrastructure-Foundations, netops, SRE
cmooney added a comment to T321704: Netbox network report failing - timeout errors.

I've took a look today and trying to manually run all the tests there isn't anyone that takes so long to trigger the 300s timeout, not even summing all of them.

Mon, Jun 10, 11:29 AM · Patch-For-Review, Infrastructure-Foundations, SRE, netops

Fri, Jun 7

cmooney updated the task description for T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches.
Fri, Jun 7, 7:59 PM · netops, Infrastructure-Foundations, SRE
cmooney updated the task description for T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches.
Fri, Jun 7, 7:52 PM · netops, Infrastructure-Foundations, SRE
cmooney closed T365169: Switch BGP (EVPN) topology between rows/spines at core sites as Resolved.

Folk seem happy enough with this approach so I'll close this, automation has been added and working.

Fri, Jun 7, 7:52 PM · netops, Infrastructure-Foundations, SRE
cmooney closed T365169: Switch BGP (EVPN) topology between rows/spines at core sites, a subtask of T364095: Codfw row C/D switch installation & configuration, as Resolved.
Fri, Jun 7, 7:52 PM · DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE
cmooney added a parent task for T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches: T364095: Codfw row C/D switch installation & configuration.
Fri, Jun 7, 7:50 PM · netops, Infrastructure-Foundations, SRE
cmooney added a subtask for T364095: Codfw row C/D switch installation & configuration: T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches.
Fri, Jun 7, 7:50 PM · DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE
cmooney triaged T366941: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches as Medium priority.
Fri, Jun 7, 7:49 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T360789: codfw row C/D upgrade racking task.

I tried to bring the ssw1-d8-codfw link to cr2-codfw up, but it doesn't look ready? Ticked above, but checking on cr2-codfw there is no QSFP in 1/0/2 after the bounce. Is the cabling in place for that? Still set to planned in Netbox so maybe I jumped the gun:

Fri, Jun 7, 5:40 PM · SRE, Infrastructure-Foundations, netops, ops-codfw, DC-Ops
cmooney updated subscribers of T364095: Codfw row C/D switch installation & configuration.
Fri, Jun 7, 5:15 PM · DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE

Thu, Jun 6

cmooney closed T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 as Resolved.

All work complete. Both switches seem stable after the reboot. Moving the links around worked well, big thanks to @Jclark-ctr on site for the help on that.

Thu, Jun 6, 5:51 PM · netops, Infrastructure-Foundations, SRE
cmooney closed T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3, a subtask of T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2, as Resolved.
Thu, Jun 6, 5:50 PM · Infrastructure-Foundations, netops, SRE
cmooney added a comment to T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

ssw1-f1-eqiad has now been upgraded to 22.2R3 also, no issues to report.

Thu, Jun 6, 3:53 PM · netops, Infrastructure-Foundations, SRE
cmooney created P64200 (An Untitled Masterwork).
Thu, Jun 6, 3:48 PM
cmooney added a comment to T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

The first phase of this is complete, ssw1-e1-eqiad has been upgraded.

Thu, Jun 6, 1:16 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

Detailed steps are in P64182

Thu, Jun 6, 11:37 AM · netops, Infrastructure-Foundations, SRE
cmooney edited P64182 Spine switch eqiad upgrade steps.
Thu, Jun 6, 11:35 AM
cmooney created P64182 Spine switch eqiad upgrade steps.
Thu, Jun 6, 11:30 AM

Wed, Jun 5

cmooney added a comment to T360789: codfw row C/D upgrade racking task.

@cmooney all good on lsw1-d4, lsw1-c2 and lsw1-d8

Wed, Jun 5, 7:44 PM · SRE, Infrastructure-Foundations, netops, ops-codfw, DC-Ops
cmooney added a comment to T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

I spoke to @Jclark-ctr earlier, we will do this commencing at 12:00 UTC tomorrow Thurs 6th Jun.

Wed, Jun 5, 7:17 PM · netops, Infrastructure-Foundations, SRE
cmooney claimed T365998: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad .
Wed, Jun 5, 7:16 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365997: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad .
Wed, Jun 5, 7:13 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T365996: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad .
Wed, Jun 5, 7:13 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365996: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad .
Wed, Jun 5, 7:12 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365995: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad.
Wed, Jun 5, 7:11 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365994: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad.
Wed, Jun 5, 7:09 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T365995: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad.
Wed, Jun 5, 7:08 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T365993: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad.
Wed, Jun 5, 7:06 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365993: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad.
Wed, Jun 5, 7:06 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney added a comment to T365988: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad.

From the swift POV, this is just checking the cluster is happy afterwards. I'm on annual leave when this is happening, though, so we'll need to find someone to cover once we know what time the work is planned for.

Wed, Jun 5, 7:04 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T365988: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad.
Wed, Jun 5, 7:03 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365988: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad.
Wed, Jun 5, 7:03 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365987: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad.
Wed, Jun 5, 7:01 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365986: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad.
Wed, Jun 5, 7:01 PM · Data-Platform-SRE, SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney claimed T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad .
Wed, Jun 5, 6:58 PM · SRE-swift-storage, DBA, Data-Persistence, Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.
Wed, Jun 5, 2:32 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

@cmooney as it turns out, I will be out until June 10th.

Wed, Jun 5, 10:41 AM · netops, Infrastructure-Foundations, SRE
cmooney updated subscribers of T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.

@Jclark-ctr @VRiley-WMF unfortunately these switch upgrades require us to shift some cables around before/after the upgrade to avoid disrupting services.

Wed, Jun 5, 10:23 AM · netops, Infrastructure-Foundations, SRE
cmooney updated the task description for T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.
Wed, Jun 5, 10:18 AM · netops, Infrastructure-Foundations, SRE
cmooney updated the task description for T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.
Wed, Jun 5, 8:49 AM · netops, Infrastructure-Foundations, SRE

Tue, Jun 4

cmooney updated the task description for T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.
Tue, Jun 4, 6:28 PM · netops, Infrastructure-Foundations, SRE
cmooney created P64030 (An Untitled Masterwork).
Tue, Jun 4, 6:21 PM
cmooney updated the task description for T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2.
Tue, Jun 4, 6:19 PM · Infrastructure-Foundations, netops, SRE
cmooney updated the task description for T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.
Tue, Jun 4, 6:19 PM · netops, Infrastructure-Foundations, SRE
cmooney added a comment to T366193: Anycast ns1.wikimedia.org.

IMHO, the A/B set solution with a pair of anycasts, is the most elegant and simple way to achieve the best balance of resiliency and perf for our authdns.

Tue, Jun 4, 10:35 AM · SRE, Traffic