Change Details

**Devices**: ssw1-e1-eqiad, ssw1-f1-eqiad **When**: Thur Jun 5th 2024, 15:00 UTC **Downtime**: 60 minutes As discussed in the parent task we need to upgrade all our QFX5120 devices in Ashburn to a more recent JunOS to overcome some bugs and use newer features. This Spine-layer switches can mostly be done without affecting servers. The top-of-rack switches in each rack have links to both Spines, so we can upgrade one at a time without cutting comms to any rack. However we do have connections from LVS servers in remote rows (A-D) which land on the spine switches so the load-balancers can reach backend servers in rows E/F. (The list of hosts/services with backends in these rows can is here: P63779) The LVS servers are connected as follows: |Host|Interface|Switch| |-------|--------------|----------| |lvs1017|enp94s0f0np0|ssw1-e1-eqiad| |lvs1018|enp94s0f0np0|ssw1-e1-eqiad| |lvs1019|enp94s0f0np0|ssw1-f1-eqiad| |lvs1020|enp94s0f0np0|ssw1-f1-eqiad| lvs1020 is the backup LVS server for all the others. That means we cannot upgrade ssw1-f1-eqiad without a complete outage to all services fronted by lvs1019, as the reboot will also disrupt comms to rows E/F from the only backup, lvs1020. While it might be possible to do ssw1-e1-eqiad by failing both lvs1017 and lvs1018 over to lvs1020 in advance, Traffic advise it is not a good idea to have all the requests those hosts handle re-routed to the single backup host. So in both cases to complete the work we will need to depool eqiad in DNS, to stop traffic being sent to the LVS VIPs there, which will mean the break in connectivity from LVS hosts to rows E and F won't be an issue. Given that is the case we will plan to upgrade both Spine switches one after another, so both are done during a single window/depool. Plan would be to **depool eqiad at 14:00 UTC**, giving some time for DNS changes to take affect before starting the **first switch update at 15:00 UTC** and then proceeding to the next. Each switch upgrade is expected to take between 20-30 mintues to complete, after which we can verify things look good and repool the site.

**Devices**: ssw1-e1-eqiad, ssw1-f1-eqiad **When**: Thur Jun 6th 2024, 15:00 UTC **Downtime**: 60 minutes As discussed in the parent task we need to upgrade all our QFX5120 devices in Ashburn to a more recent JunOS to overcome some bugs and use newer features. This Spine-layer switches can mostly be done without affecting servers. The top-of-rack switches in each rack have links to both Spines, so we can upgrade one at a time without cutting comms to any rack. However we do have connections from LVS servers in remote rows (A-D) which land on the spine switches so the load-balancers can reach backend servers in rows E/F. (The list of hosts/services with backends in these rows can is here: P63779) The LVS servers are connected as follows: |Host|Interface|Switch| |-------|--------------|----------| |lvs1017|enp94s0f0np0|ssw1-e1-eqiad| |lvs1018|enp94s0f0np0|ssw1-e1-eqiad| |lvs1019|enp94s0f0np0|ssw1-f1-eqiad| |lvs1020|enp94s0f0np0|ssw1-f1-eqiad| lvs1020 is the backup LVS server for all the others. That means we cannot upgrade ssw1-f1-eqiad without a complete outage to all services fronted by lvs1019, as the reboot will also disrupt comms to rows E/F from the only backup, lvs1020. While it might be possible to do ssw1-e1-eqiad by failing both lvs1017 and lvs1018 over to lvs1020 in advance, Traffic advise it is not a good idea to have all the requests those hosts handle re-routed to the single backup host. So in both cases to complete the work we will need to depool eqiad in DNS, to stop traffic being sent to the LVS VIPs there, which will mean the break in connectivity from LVS hosts to rows E and F won't be an issue. Given that is the case we will plan to upgrade both Spine switches one after another, so both are done during a single window/depool. Plan would be to **depool eqiad at 14:00 UTC**, giving some time for DNS changes to take affect before starting the **first switch update at 15:00 UTC** and then proceeding to the next. Each switch upgrade is expected to take between 20-30 mintues to complete, after which we can verify things look good and repool the site.

**Devices**: ssw1-e1-eqiad, ssw1-f1-eqiad **When**: Thur Jun 56th 2024, 15:00 UTC **Downtime**: 60 minutes As discussed in the parent task we need to upgrade all our QFX5120 devices in Ashburn to a more recent JunOS to overcome some bugs and use newer features. This Spine-layer switches can mostly be done without affecting servers. The top-of-rack switches in each rack have links to both Spines, so we can upgrade one at a time without cutting comms to any rack. However we do have connections from LVS servers in remote rows (A-D) which land on the spine switches so the load-balancers can reach backend servers in rows E/F. (The list of hosts/services with backends in these rows can is here: P63779) The LVS servers are connected as follows: |Host|Interface|Switch| |-------|--------------|----------| |lvs1017|enp94s0f0np0|ssw1-e1-eqiad| |lvs1018|enp94s0f0np0|ssw1-e1-eqiad| |lvs1019|enp94s0f0np0|ssw1-f1-eqiad| |lvs1020|enp94s0f0np0|ssw1-f1-eqiad| lvs1020 is the backup LVS server for all the others. That means we cannot upgrade ssw1-f1-eqiad without a complete outage to all services fronted by lvs1019, as the reboot will also disrupt comms to rows E/F from the only backup, lvs1020. While it might be possible to do ssw1-e1-eqiad by failing both lvs1017 and lvs1018 over to lvs1020 in advance, Traffic advise it is not a good idea to have all the requests those hosts handle re-routed to the single backup host. So in both cases to complete the work we will need to depool eqiad in DNS, to stop traffic being sent to the LVS VIPs there, which will mean the break in connectivity from LVS hosts to rows E and F won't be an issue. Given that is the case we will plan to upgrade both Spine switches one after another, so both are done during a single window/depool. Plan would be to **depool eqiad at 14:00 UTC**, giving some time for DNS changes to take affect before starting the **first switch update at 15:00 UTC** and then proceeding to the next. Each switch upgrade is expected to take between 20-30 mintues to complete, after which we can verify things look good and repool the site.