Page MenuHomePhabricator

Upgrade routers
Open, LowPublic

Description

hostnamecurrent versiontarget version
cr1-eqsin15.1R7.817.3R3-S7
cr2-esams13.3R8.717.3R3-S7
cr3-esams15.1F5.1517.3R3-S7

In combination with T242947#5812926

Details

Related Gerrit Patches:
operations/dns : masterDepool eqsin for router upgrade

Event Timeline

ayounsi triaged this task as Low priority.Jan 17 2020, 3:16 PM
ayounsi created this task.
ayounsi created this object with edit policy "acl*sre-team (Project)".
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2020, 3:16 PM
ayounsi updated the task description. (Show Details)Jan 17 2020, 4:06 PM
ayounsi added a comment.EditedFri, Feb 7, 3:53 PM

Current dates are:
Feb. 11th - 21:00UTC - 1h - cr1-eqsin - eqsin will be depooled (this is when eqsin sees the less traffic) - upgrade doc
Feb. 12th - 13:00UTC - 2h - cr2/3-esams

Change 571368 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool eqsin for router upgrade

https://gerrit.wikimedia.org/r/571368

Change 571368 merged by Ayounsi:
[operations/dns@master] Depool eqsin for router upgrade

https://gerrit.wikimedia.org/r/571368

Mentioned in SAL (#wikimedia-operations) [2020-02-11T20:08:08Z] <XioNoX> depool eqsin for router upgrade - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-11T20:51:31Z] <XioNoX> reboot backup RE on cr1-eqsin - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-11T21:32:43Z] <XioNoX> switchover RE mastership on cr1-eqsin - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-11T21:50:55Z] <XioNoX> reboot re0:cr1-eqsin (backup) - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-11T22:04:10Z] <XioNoX> switchover RE mastership back re0 on cr1-eqsin - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-12T00:33:15Z] <XioNoX> commit full on cr1-eqsin - T243080

ayounsi added a comment.EditedWed, Feb 12, 1:05 AM

cr1-eqsin RPD keeps crashing and core-dumping since the upgrade see T244944

ayounsi updated the task description. (Show Details)Wed, Feb 12, 2:35 PM

cr1-eqsin is back to normal, next step is to plan esams.

Feb. 18th - 13:00UTC - 2h - cr2/3-esams

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:36:15Z] <XioNoX> push new Junos to cr2-esams:re1 (backup RE, noop) - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:45:33Z] <XioNoX> remove graceful-switchover and nonstop-routing from cr2-esams - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:10:12Z] <XioNoX> fail vrrp master to cr3-esams - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:23:06Z] <XioNoX> bump cost of eqiad-esams transport - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:24:15Z] <XioNoX> reboot cr2-esams:re1 (backup) - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:37:39Z] <XioNoX> deactivate peering/transit on cr2-esams - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:39:22Z] <XioNoX> cr2-esams - request chassis routing-engine master switch - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T13:44:28Z] <XioNoX> installing OS on cr2-esams:re0 - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:01:25Z] <XioNoX> re-enable cr2-esams BGP group IX4 - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:29:12Z] <XioNoX> re-disable cr2-esams BGP group IX4 - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:31:24Z] <XioNoX> cr2-esams - request chassis routing-engine master switch - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:33:23Z] <XioNoX> re-enable cr2-esams BGP transit/peering - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:34:40Z] <XioNoX> restore default esams-eqiad link cost - T243080

Mentioned in SAL (#wikimedia-operations) [2020-02-18T14:39:09Z] <XioNoX> remove cr2-esams VRRP handicap - T243080