Page MenuHomePhabricator
Create Task
Maniphest T226422

update RE-S-X6-64G-S in cr[12]-codfw
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This task will track the installation of (4) new RE-S-X6-64G-S modules for installation/swap (2 in each) in cr[12]-codfw.

This work will require coordination with the netops team for scheduling and execution.

  • Receive in and update procurement task T218794
  • Schedule work with @ayounsi and update this checklist with other steps.

Window steps:

  • Downtime alerting (Icinga/Librenms)
  • Ensure VRRP master is on the other node
  • Tune OSPF cost to drain transport links terminating on that device
  • Drain local BGP peers (graceful shutdown + deactivate)
  • Fail LVS over the ones connected to the other router
  • Ensure RE is connected to serial console
  • From the doc do Removing the Routing Engine and Installing the Routing Engine RE-S-X6-64G for the backup routing engine
  • Warn on IRC that it's going to be bumpy
  • From the doc do Verifying and Configuring the Upgraded Routing Engine as the Master All FPCs reboot after this step.
  • Verify device is healthy (logs, OSPF/BGP sessions, alarms, alerting)
  • Do the 5 steps above for the other RE (now backup)
  • Rollback BGP/OSPF/LVS/VRRP changes

Details

SubjectRepoBranchLines +/-
Depool codfw and eqsin for codfw routers workoperations/dnsmaster+3 -0
Varnish: redirect eqsin/ulsfo text to eqiadoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedayounsiT196432 Configure interface damping on primary links
 ResolvedayounsiT226422 update RE-S-X6-64G-S in cr[12]-codfw

Event Timeline

RobH triaged this task as Medium priority.Jun 24 2019, 3:59 PM
RobH created this task.
Restricted Application added a project: SRE. · View Herald TranscriptJun 24 2019, 3:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Jun 24 2019, 3:59 PM
ayounsi updated the task description. (Show Details)Jun 25 2019, 7:04 AM
Papaul added a comment.Jul 18 2019, 10:03 PM

@ayounsi please pick a day and time sometimes next week (Mon - Wed) and let me know.

Thanks

ayounsi added a comment.Jul 24 2019, 8:33 PM

Scheduled for the 31st at 15:00UTC (1h total).

Papaul updated the task description. (Show Details)Jul 31 2019, 2:49 PM
Stashbot added a comment.Jul 31 2019, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-31T15:03:51Z] <XioNoX> power down re1:cr1-codfw (backup) - T226422

ayounsi added a comment.Jul 31 2019, 4:21 PM

The new backup routing engine is not coming online.
Rolling back to the old one is not working neither.
Opened JTAC Service Request ID: 2019-0731-0446 .

ayounsi added a comment.Jul 31 2019, 8:34 PM

First JTAC suggestion is to re-seat the SCB. We didn't do that today as the doc wasn't clear if it could be done with the router online.
JTAC is looking into the logs.

ayounsi added a comment.Jul 31 2019, 8:38 PM

Also noticed the following while looking at the doc again today:

  1. Use the request chassis routing-engine master switch command to make the Routing Engine RE-S-X6-64G (RE1) the master Routing Engine. All FPCs reboot after this step.

So this mean the RE failover will have production impact, and should be prepared accordingly.

Stashbot added a comment.Aug 8 2019, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-08T14:52:00Z] <XioNoX> continue cr1-codfw:re1 replacement - T226422

Stashbot added a comment.Aug 8 2019, 3:11 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-08T15:11:03Z] <XioNoX> commit synchronize on cr1-codfw - T226422

Papaul added a comment.Aug 8 2019, 4:13 PM

Return information for bad SCB

New Doc 2019-08-08 10.58.07_1.pdf422 KBDownload

ayounsi updated the task description. (Show Details)Aug 15 2019, 8:58 PM
gerritbot added a comment.Aug 21 2019, 2:33 PM

Change 531502 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool codfw for routers work

https://gerrit.wikimedia.org/r/531502

gerritbot added a project: Patch-For-Review.Aug 21 2019, 2:33 PM
gerritbot added a comment.Aug 21 2019, 3:09 PM

Change 531513 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Varnish: redirect eqsin/ulsfo text to eqiad

https://gerrit.wikimedia.org/r/531513

gerritbot added a comment.Aug 21 2019, 4:36 PM

Change 531502 merged by Ayounsi:
[operations/dns@master] Depool codfw and eqsin for codfw routers work

https://gerrit.wikimedia.org/r/531502

Stashbot added a comment.Aug 21 2019, 4:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T16:37:04Z] <XioNoX> depool eqsin and codfw - T226422

Stashbot added a comment.Aug 21 2019, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T16:46:26Z] <XioNoX> apply BGP graceful shutdown to cr1-codfw transits - T226422

Stashbot added a comment.Aug 21 2019, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T16:51:12Z] <XioNoX> increase OSPF cost on ulsfo-codfw link - T226422

gerritbot added a comment.Aug 21 2019, 4:56 PM

Change 531513 merged by Ayounsi:
[operations/puppet@production] Varnish: redirect eqsin/ulsfo text to eqiad

https://gerrit.wikimedia.org/r/531513

Stashbot added a comment.Aug 21 2019, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T16:56:22Z] <XioNoX> Varnish: redirect eqsin/ulsfo text to eqiad - T226422

Stashbot added a comment.Aug 21 2019, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:08:47Z] <XioNoX> disable BGP from cr1-codfw to lvs2001/2/3 - T226422

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2019, 5:11 PM
Stashbot added a comment.Aug 21 2019, 5:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:17:11Z] <XioNoX> failover master RE to RE1 on cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 5:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:25:50Z] <XioNoX> shutdown RE0 on cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:33:33Z] <XioNoX> failover master RE to RE0 on cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:43:02Z] <XioNoX> restart both REs on cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 5:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:53:23Z] <XioNoX> rollback: disable BGP from cr1-codfw to lvs2001/2/3 - T226422

Stashbot added a comment.Aug 21 2019, 5:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:55:03Z] <XioNoX> Rollback: increase OSPF cost on ulsfo-codfw link - T226422

Stashbot added a comment.Aug 21 2019, 5:56 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T17:56:04Z] <XioNoX> rollback: apply BGP graceful shutdown to cr1-codfw transits - T226422

Stashbot added a comment.Aug 21 2019, 6:04 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:04:32Z] <XioNoX> increase OSPF cost on cr2-codfw links - T226422

Stashbot added a comment.Aug 21 2019, 6:12 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:12:25Z] <XioNoX> deactivate transit links on cr2-codfw - T226422

Stashbot added a comment.Aug 21 2019, 6:18 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:17:53Z] <XioNoX> move VRRP master from cr2-codfw to cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 6:19 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:19:26Z] <XioNoX> shutdown re1:cr2-codfw (backup) - T226422

Stashbot added a comment.Aug 21 2019, 6:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:32:23Z] <XioNoX> failover master RE to RE1 on cr2-codfw - T226422

Stashbot added a comment.Aug 21 2019, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T18:37:03Z] <XioNoX> shutdown re0:cr2-codfw (backup) - T226422

Stashbot added a comment.Aug 21 2019, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:14:13Z] <XioNoX> failover master RE to RE0 on cr2-codfw - T226422

Stashbot added a comment.Aug 21 2019, 7:16 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:16:25Z] <XioNoX> restart both REs on cr2-codfw - T226422

Stashbot added a comment.Aug 21 2019, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:24:53Z] <XioNoX> rollback: move VRRP master from cr2-codfw to cr1-codfw - T226422

Stashbot added a comment.Aug 21 2019, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:25:40Z] <XioNoX> rollback deactivate transit links on cr2-codfw - T226422

Stashbot added a comment.Aug 21 2019, 7:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:26:14Z] <XioNoX> rollback: increase OSPF cost on cr2-codfw links - T226422

Stashbot added a comment.Aug 21 2019, 7:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:31:41Z] <XioNoX> Rollback: Varnish: redirect eqsin/ulsfo text to eqiad - T226422

Stashbot added a comment.Aug 21 2019, 7:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-21T19:34:38Z] <XioNoX> repool codfw and eqsin - T226422

ayounsi closed this task as Resolved.Aug 21 2019, 7:37 PM

DONE!
Everything is healthy, very little alert noise, no service impact.

ayounsi added a parent task: T196432: Configure interface damping on primary links.Aug 21 2019, 10:52 PM
ayounsi mentioned this in T211728: Outbound BGP graceful shutdown.Sep 24 2019, 9:40 PM
ayounsi mentioned this in T305126: Make more extensive use of Netbox custom fields.Jul 18 2022, 9:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL