Upgrade core routers to Junos 21+
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	ayounsi
	Nov 15 2021, 2:50 PM

Description

The Junos 21 branch is now in Junos recommended versions for MX routers, see https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476&smlogin=true

Upgrading our MXs has several advantages:

Testing routers redundancy in a planned window
Keeping a tight Junos version spread (we currently have 12, 14, 15, 17, 18, 20)
Leveraging features improvements (eg. DNS in mgmt-junos see T269340)
Fixing low risk security issues

Process is documented in https://wikitech.wikimedia.org/wiki/Juniper_router_upgrade

Device	Scheduled for	Status
cr3-ulsfo	2022-09-06 - 08:00 UTC - 2h	T295690#8213789
cr4-ulsfo	2022-09-06 - 08:00 UTC - 2h	T295690#8213789
cr3-eqsin	2022-09-07 - 08:00 UTC - 2h	T295690#8217086
cr2-eqsin	2022-09-07 - 08:00 UTC - 2h	T295690#8217086
cr3-knams	2022-09-08 - 08:00 UTC - 2h	T295690#8220460
cr2-esams	2022-09-08 - 08:00 UTC - 2h	T295690#8220460
cr3-esams	2022-09-08 - 08:00 UTC - 2h	T295690#8220460
cr3-esams (attempt #2)	2022-09-12 - 08:00 UTC - 3h	T295690#8229042
cr1-codfw	2022-09-13 - 08:00 UTC - 3h	T295690#8232607
cr2-codfw	2022-09-13 - 08:00 UTC - 3h	T295690#8232607
cr2-eqdfw	2022-09-14 - 08:00 UTC - 1h	T295690#8236185
cr1-eqiad	2022-09-29 - 08:00 UTC - 3h	T295690#8272177
cr2-eqiad	2022-09-29 - 08:00 UTC - 3h	T295690#8272177
cr2-eqord	2022-09-29 - 08:00 UTC - 3h	T295690#8272177

Then remove unnecessary images from apt1001:/srv/junos/

Details

Project	Branch	Lines +/-	Subject
operations/homer/public	master	+0 -13	Fully remove VRRP auth
operations/dns	master	+0 -2	Re-pool codfw after upgrading core routers on site
operations/homer/public	master	+1 -0	Disable VRRP auth between CRs in codfw
operations/dns	master	+2 -0	Depool codfw prior to core router upgrades.
operations/dns	master	+0 -2	Repool esams after cr3-esams core router upgrade.
operations/dns	master	+2 -0	Depool esams for cr3-esams core router upgrade.
operations/dns	master	+3 -0	Depool esams for routers upgrades
operations/homer/public	master	+1 -0	Disable VRRP auth for esams
operations/dns	master	+0 -2	Revert "Depool eqsin for core router upgrades."
operations/homer/public	master	+1 -0	Disbale VRRP auth in eqsin
operations/dns	master	+2 -0	Depool eqsin for core router upgrades.
operations/dns	master	+3 -0	Depool ulsfo for routers ugprades
operations/homer/public	master	+8 -0	Add variable to disable VRRP auth

Related Objects
Search...

Status	Assigned	Task
Open	None	T253824 planned upstream deprecation of the ssh-rsa signing algorithm (RSA with SHA-1)
Resolved	ayounsi	T254013 all network devices must run OpenSSH >= 7.2p1 but != 7.4p1
Resolved	ayounsi	T317175 Junos: resolve DNS through mgmt_junos
Resolved	ayounsi	T327862 Use mgmt_junos on all network devices
		Restricted Task
Open	None	T316539 Upgrade network devices to Junos 20+
Resolved	None	T295690 Upgrade core routers to Junos 21+

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

ayounsi updated the task description. (Show Details)Aug 29 2022, 12:30 PM

ayounsi mentioned this in T316529: Upgrade management routers and switches to Junos 21.Aug 29 2022, 12:46 PM

ayounsi mentioned this in T316532: Upgrade POPs asw to Junos 21.Aug 29 2022, 1:03 PM

ayounsi added a parent task: T316539: Upgrade network devices to Junos 20+.Aug 29 2022, 1:27 PM

cmooney updated the task description. (Show Details)Sep 2 2022, 9:00 AM

ayounsi updated the task description. (Show Details)Sep 6 2022, 6:14 AM

Change 830085 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Depool ulsfo for routers ugprades

https://gerrit.wikimedia.org/r/830085

gerritbot added a project: Patch-For-Review.Sep 6 2022, 7:44 AM

Change 830085 merged by Ayounsi:

[operations/dns@master] Depool ulsfo for routers ugprades

https://gerrit.wikimedia.org/r/830085

Mentioned in SAL (#wikimedia-operations) [2022-09-06T07:52:36Z] <XioNoX> depool ulsfo for routers upgrade - T295690

Icinga downtime and Alertmanager silence (ID=7eb8120c-f8b6-4c79-8deb-b18a305a2353) set by ayounsi@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: router upgrade

cr3-ulsfo.wikimedia.org

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2022, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2022-09-06T08:42:10Z] <XioNoX> restart cr3-ulsfo for software upgrade - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-06T10:26:38Z] <XioNoX> put cr3-ulsfo back in service - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-06T10:42:34Z] <XioNoX> drain traffic from cr4-ulsfo - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-06T11:06:31Z] <XioNoX> restart cr4-ulsfo for software upgrade - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-06T11:17:58Z] <XioNoX> put cr4-ulsfo back in service - T295690

This has been quite eventful.

To keep in mind that those upgrade need the no-validate knob, more details in the doc

We first went for Junos 21.4R2-Sx which is the most recent Junos recommended version, but there is an annoying bug preventing the FPC to come online.
We then downgraded to Junos 21.2R2-Sx. There the FPC worked fine, but there are 2 issues:

VRRP adjacency to (at least) older Junos, using MD5 keys don't establish, causing a split brain (both routers are master), solved by remove the md5 key as it's only present on v4 and doesn't bring much benefits
On cr3-ulsfo some BFD sessions are still down (but upper BGP is fine)

Change 830156 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Add variable to disable VRRP auth

https://gerrit.wikimedia.org/r/830156

gerritbot added a project: Patch-For-Review.Sep 6 2022, 12:04 PM

ayounsi updated the task description. (Show Details)Sep 6 2022, 12:10 PM

Change 830156 merged by Ayounsi:

[operations/homer/public@master] Add variable to disable VRRP auth

https://gerrit.wikimedia.org/r/830156

Mentioned in SAL (#wikimedia-operations) [2022-09-06T12:15:57Z] <XioNoX> repool ulsfo - T295690

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2022, 12:30 PM

Change 830499 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Depool eqsin for core router upgrades.

https://gerrit.wikimedia.org/r/830499

gerritbot added a project: Patch-For-Review.Sep 7 2022, 7:35 AM

Change 830499 merged by Cathal Mooney:

[operations/dns@master] Depool eqsin for core router upgrades.

https://gerrit.wikimedia.org/r/830499

Mentioned in SAL (#wikimedia-operations) [2022-09-07T07:46:16Z] <topranks> Depool eqsin from user traffic in advance of core router upgrades - T295690

Change 830563 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Disbale VRRP auth in eqsin

https://gerrit.wikimedia.org/r/830563

Change 830563 merged by jenkins-bot:

[operations/homer/public@master] Disbale VRRP auth in eqsin

https://gerrit.wikimedia.org/r/830563

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2022, 8:30 AM

Icinga downtime and Alertmanager silence (ID=7af287ca-21ab-4f9d-adb3-478641fdd465) set by cmooney@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: router upgrade

cr2-eqsin

Icinga downtime and Alertmanager silence (ID=826e80d5-55a6-4bb6-ab1c-e094eba7f6cd) set by cmooney@cumin1001 for 1:00:00 on 1 host(s) and their services with reason: router upgrade

cr3-eqsin

Change 830575 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Revert "Depool eqsin for core router upgrades."

https://gerrit.wikimedia.org/r/830575

gerritbot added a project: Patch-For-Review.Sep 7 2022, 9:41 AM

Change 830575 merged by Cathal Mooney:

[operations/dns@master] Revert "Depool eqsin for core router upgrades."

https://gerrit.wikimedia.org/r/830575

Mentioned in SAL (#wikimedia-operations) [2022-09-07T09:48:09Z] <topranks> Re-pooling eqsin for user traffic after successful core router upgrades - T295690

Upgrade completed ok for cr2-eqsin and cr3-eqsin.

Went straight to 21.2R3-S2.9 based on experience in ulsfo, all went ok. Used no-validate when adding image to device.

No issues encountered, BFD adjacencies formed ok after reboot on both.

cmooney updated the task description. (Show Details)Sep 7 2022, 11:13 AM

Change 830729 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Depool esams for routers upgrades

https://gerrit.wikimedia.org/r/830729

Change 830730 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Disable VRRP auth for esams

https://gerrit.wikimedia.org/r/830730

Change 830729 merged by Ayounsi:

[operations/dns@master] Depool esams for routers upgrades

https://gerrit.wikimedia.org/r/830729

Mentioned in SAL (#wikimedia-operations) [2022-09-08T07:41:06Z] <XioNoX> depool esams for routers upgrade - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-08T08:07:28Z] <XioNoX> drain draffic from cr3-esams - T295690

Icinga downtime and Alertmanager silence (ID=3b336fa4-f522-4b10-abdb-d6be83f6a04a) set by ayounsi@cumin2002 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr3-esams,cr3-esams IPv6,re0.cr3-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2022-09-08T08:44:27Z] <XioNoX> reverting cr3-esams changes (JTAC will be needed for a firmware upgrade), and moving on to cr2-esams - T295690

Icinga downtime and Alertmanager silence (ID=e0d9eb2b-5520-4f80-912e-3627c94e9982) set by ayounsi@cumin2002 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr2-esams,cr2-esams IPv6,re0.cr2-esams.mgmt

Icinga downtime and Alertmanager silence (ID=ff1db65d-a6ee-4e20-ae07-837bbe264b2f) set by ayounsi@cumin2002 for 2:00:00 on 2 host(s) and their services with reason: router upgrade

cr3-knams,cr3-knams IPv6

Mentioned in SAL (#wikimedia-operations) [2022-09-08T09:35:48Z] <XioNoX> drain draffic from cr3-knams - T295690

Change 830730 merged by Ayounsi:

[operations/homer/public@master] Disable VRRP auth for esams

https://gerrit.wikimedia.org/r/830730

cr2-esams and cr3-knams got upgraded as expected.
cr3-esams failed as it requires a firmware upgrade, and only JTAC can provide us the firmware. We will follow up with them.

Also request system storage cleanup re1 has been added to the doc for multi-RE devices.

ayounsi updated the task description. (Show Details)Sep 8 2022, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2022-09-08T10:07:42Z] <XioNoX> re-pool esams after routers upgrade - T295690

The firmware provided by Juniper seems to be accepted by cr3-esams:

cmooney@re0.cr3-esams> show system firmware | match "^Part|version|i40"
Part             Type           Tag Current   Available Status
                                    version   version
Routing Engine 0 RE i40e-NVM    7   4.26      6.01      OK

I've scheduled this for 08:00 UTC on Monday to try again.

cmooney updated the task description. (Show Details)Sep 8 2022, 10:52 AM

Change 831479 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Depool esams for cr3-esams core router upgrade.

https://gerrit.wikimedia.org/r/831479

Change 831479 merged by Cathal Mooney:

[operations/dns@master] Depool esams for cr3-esams core router upgrade.

https://gerrit.wikimedia.org/r/831479

Mentioned in SAL (#wikimedia-operations) [2022-09-12T08:00:38Z] <topranks> de-pooliong esams in advance of upgrade to core router cr3-esams T295690

Icinga downtime and Alertmanager silence (ID=1e573369-5fdd-4621-8ae7-786b5a67de04) set by cmooney@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: router upgrade

cr3-esams

Icinga downtime and Alertmanager silence (ID=57f0ae1d-0fa1-4b98-9454-bea638ac3971) set by cmooney@cumin1001 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr3-esams,cr3-esams IPv6,re0.cr3-esams.mgmt

Icinga downtime and Alertmanager silence (ID=39465e0b-b93d-45ba-b1d8-0c49dacc39fb) set by cmooney@cumin1001 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr3-esams,cr3-esams IPv6,re0.cr3-esams.mgmt

Change 831506 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Repool esams after cr3-esams core router upgrade.

https://gerrit.wikimedia.org/r/831506

Change 831506 merged by Cathal Mooney:

[operations/dns@master] Repool esams after cr3-esams core router upgrade.

https://gerrit.wikimedia.org/r/831506

Mentioned in SAL (#wikimedia-operations) [2022-09-12T10:55:10Z] <topranks> re-pooliong esams after successful upgrade of core router cr3-esams T295690

Upgrade of cr3-esams went well earlier. Firmware upgrade works as per docs. I will put up more info on that later for our own reference.

cmooney updated the task description. (Show Details)Sep 12 2022, 1:50 PM

cmooney updated the task description. (Show Details)

Change 831800 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Depool codfw prior to core router upgrades.

https://gerrit.wikimedia.org/r/831800

Change 831800 merged by Cathal Mooney:

[operations/dns@master] Depool codfw prior to core router upgrades.

https://gerrit.wikimedia.org/r/831800

Icinga downtime and Alertmanager silence (ID=927fadc1-f5b2-478f-95ce-98bfc47881a9) set by cmooney@cumin1001 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt

Change 831840 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Disable VRRP auth between CRs in codfw

https://gerrit.wikimedia.org/r/831840

Change 831840 merged by jenkins-bot:

[operations/homer/public@master] Disable VRRP auth between CRs in codfw

https://gerrit.wikimedia.org/r/831840

Change 831889 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Re-pool codfw after upgrading core routers on site

https://gerrit.wikimedia.org/r/831889

Change 831889 merged by Cathal Mooney:

[operations/dns@master] Re-pool codfw after upgrading core routers on site

https://gerrit.wikimedia.org/r/831889

cr1-codfw and cr2-codfw sucessfully upgraded today. Took a while with the firmware upgrades too, I've added some notes here on probably the best way to approach upgrading both the firmware and JunOS with fewest switchovers.

Given the time I'll re-schedule cr2-eqdfw for tomorrow morning (Wed 14 Sept) instead.

cmooney updated the task description. (Show Details)Sep 13 2022, 2:41 PM

cr2-eqdfw upgrade completed successfully today.

cmooney updated the task description. (Show Details)Sep 14 2022, 1:42 PM

Icinga downtime and Alertmanager silence (ID=1ea26f52-695b-41ae-a3b4-28808d44161a) set by ayounsi@cumin1001 for 4:00:00 on 3 host(s) and their services with reason: router upgrade

cr1-eqiad,cr1-eqiad IPv6,re0.cr1-eqiad.mgmt

Mentioned in SAL (#wikimedia-operations) [2022-09-29T07:57:58Z] <XioNoX> drain traffic away from cr1-eqiad - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T08:15:51Z] <XioNoX> first cr1-eqiad RE switchover (for NVM firmware) - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T08:43:35Z] <XioNoX> second cr1-eqiad RE switchover - T295690

Change 836727 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Fully remove VRRP auth

https://gerrit.wikimedia.org/r/836727

Change 836727 merged by jenkins-bot:

[operations/homer/public@master] Fully remove VRRP auth

https://gerrit.wikimedia.org/r/836727

Mentioned in SAL (#wikimedia-operations) [2022-09-29T09:16:26Z] <XioNoX> repool cr1-eqiad - T295690

Icinga downtime and Alertmanager silence (ID=acbab0ff-4998-42b3-b0ad-a6be933dfff6) set by ayounsi@cumin1001 for 4:00:00 on 3 host(s) and their services with reason: router upgrade

cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt

Mentioned in SAL (#wikimedia-operations) [2022-09-29T09:33:29Z] <XioNoX> drain cr2-eqiad - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T09:42:38Z] <XioNoX> first cr2-eqiad RE switchover - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T10:07:44Z] <XioNoX> second (and longest) cr2-eqiad RE switchover - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T10:40:06Z] <XioNoX> repool cr2-eqiad - T295690

Icinga downtime and Alertmanager silence (ID=01f0d013-5101-4278-93a6-1ea49f9dea28) set by ayounsi@cumin1001 for 1:00:00 on 2 host(s) and their services with reason: router upgrade

cr2-eqord,cr2-eqord IPv6

Mentioned in SAL (#wikimedia-operations) [2022-09-29T10:53:31Z] <XioNoX> drain cr2-eqord - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T11:06:21Z] <XioNoX> restart cr2-eqord for upgrade - T295690

Mentioned in SAL (#wikimedia-operations) [2022-09-29T11:16:06Z] <XioNoX> re-pool cr2-eqord - T295690

eqiad and eqord went extremely well.

Thanks @cmooney for the firmware instructions

ayounsi updated the task description. (Show Details)Sep 29 2022, 11:21 AM

ayounsi closed this task as Resolved.Sep 29 2022, 11:24 AM

ayounsi updated the task description. (Show Details)

I went through the useful https://apps.juniper.net/feature-explorer/select-software.html?typ=1&swName=Junos%20OS&rel=21.2R3&sid=1211&platform=MX204&pid=11310204
All the way down to "Features Introduced in Release - Junos OS 18.1R1"

Some interesting ones:

System log (syslog control plane) over TLS with IPv4/IPv6 - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=10863&fn=System%20log%20(syslog%20control%20plane)%20over%20TLS%20with%20IPv4%2FIPv6
BGP Auto-discovered Neighbor - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=10791&fn=BGP%20Auto-discovered%20Neighbor
Bidirectional Forwarding Detection (BFD) Strict Mode for OSPF - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=9149&fn=Bidirectional%20Forwarding%20Detection%20(BFD)%20Strict%20Mode%20for%20OSPF
BGP RIB Sharding - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=9130&fn=BGP%20RIB%20Sharding
Optimized BGP peer reestablishment - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=9160&fn=Optimized%20BGP%20peer%20reestablishment
Python 3 support for commit, event, op, and SNMP scripts - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=9031&fn=Python%203%20support%20for%20commit%2C%20event%2C%20op%2C%20and%20SNMP%20scripts
BGP graceful shutdown - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=8795&fn=BGP%20graceful%20shutdown
System logging over management VRF (mgmt_junos) - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=8739&fn=System%20logging%20over%20management%20VRF%20(mgmt_junos)
Getting load-balancing hash result information - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=7685&fn=Getting%20load-balancing%20hash%20result%20information
BGP large communities - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=7963&fn=BGP%20large%20communities
BGP multipath at global level - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=8291&fn=BGP%20multipath%20at%20global%20level
Advertising Aggregate Bandwidth Across External BGP Links for Load Balancing - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=8175&fn=Advertising%20Aggregate%20Bandwidth%20Across%20External%20BGP%20Links%20for%20Load%20Balancing
sFlow technology - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=1314&fn=sFlow%20technology
ZTP – Zero Touch Provisioning (EZ Touchless Provisioning using DHCP) - https://apps.juniper.net/feature-explorer/feature-info.html?fKey=4161&fn=ZTP%20%E2%80%93%20Zero%20Touch%20Provisioning%20(EZ%20Touchless%20Provisioning%20using%20DHCP)

Upgrade core routers to Junos 21+Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Upgrade core routers to Junos 21+
Closed, ResolvedPublic
Actions

Related Objects
Search...