Page MenuHomePhabricator

Deploy cr2-eqsin
Closed, ResolvedPublic

Description

  • Figure out who is going to do the onsite work (Arzhel)
  • Decide on a date Feb. 21st/22nd
  • Verify router config is up to date with standards [Arzhel]
  • Ship router and associated cables/optics to arrive on Feb 20th latest

Onsite

  • Depool eqsin
  • Rack cr2-eqsin in U44
  • Connect mgmt/console/power
  • Verify working
  • Unplug Tata/NTT/PCCW from cr1-eqsin
  • Delete BGP config
  • Reconfigure ports:

cr1:xe-1/0/0 to cr2:xe-0/1/5
cr1:xe-2/0/0 to cr2:xe-0/1/6

cr1-eqsin
delete interfaces xe-1/0/0
delete interfaces xe-2/0/0
delete interfaces xe-2/0/2
show interfaces interface-range external member xe-1/0/0
show interfaces interface-range external member xe-2/0/0
show interfaces interface-range external member xe-2/0/2
set interfaces interface-range disabled member xe-2/0/2
set interfaces interface-range disabled description DISABLED
set interfaces interface-range disabled disable
set interfaces xe-1/0/0 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-1/0/0 gigether-options 802.3ad ae0
set interfaces xe-2/0/0 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-2/0/0 gigether-options 802.3ad ae0
set interfaces ae0 description "Core: cr2-eqsin:ae0"
set interfaces ae0 mtu 9192
set interfaces ae0 aggregated-ether-options link-speed mixed
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp periodic fast
set interfaces ae0 unit 0 family inet address 103.102.166.140/31
set interfaces ae0 unit 0 family inet6 address 2001:df2:e500:fe05::1/64
set protocols ospf area 0.0.0.0 interface ae0.0 interface-type p2p
set protocols ospf3 area 0.0.0.0 interface ae0.0 interface-type p2p
set protocols pim interface ae0.0
set protocols bgp group Confed_eqsin type internal
set protocols bgp group Confed_eqsin local-address 103.102.166.129
set protocols bgp group Confed_eqsin family inet any
set protocols bgp group Confed_eqsin family inet6 any
set protocols bgp group Confed_eqsin peer-as 65005
set protocols bgp group Confed_eqsin local-as 65005
set protocols bgp group Confed_eqsin local-as no-prepend-global-as
set protocols bgp group Confed_eqsin neighbor 103.102.166.130
  • Connect cr1<-->cr2
  • Verify link/OSPF/BGP up
  • Connect cr2:xe-0/1/3 (already configured)<-->asw:xe-0/0/20
  • Connect cr2:xe-0/1/4 (already configured)<-->asw:xe-1/0/20
cr1-eqsin
delete interfaces ae1 unit 510 family inet address 103.102.166.1/28
delete interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1::1/64
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 virtual-address 103.102.166.1
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 accept-data
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 track interface ae1.510 priority-cost 50
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:1::1
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 accept-data
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 track interface ae1.510 priority-cost 50
set interfaces ae1 unit 510 family inet6 address fe80:0:0:0:fe00::1/64

delete interfaces ae1 unit 520 family inet address 10.132.0.1/24
delete interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101::1/64
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 virtual-address 10.132.0.1
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 accept-data
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 track interface ae1.520 priority-cost 50
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:101::1
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 accept-data
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 track interface ae1.520 priority-cost 50
set interfaces ae1 unit 520 family inet6 address fe80:0:0:0:fe00::1/64

delete interfaces ae1 unit 530 family inet address 103.102.166.17/28
delete interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201::1/64
set interfaces ae1 unit 530 family inet address 103.102.166.18/28 vrrp-group 1 virtual-address 103.102.166.17
set interfaces ae1 unit 530 family inet address 103.102.166.18/28 vrrp-group 1 track interface ae1.530 priority-cost 50
set interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:201::1
set interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201:fe00::1/64 vrrp-inet6-group 1 track interface ae1.530 priority-cost 50
set interfaces ae1 unit 530 family inet6 address fe80:0:0:0:fe00::1/64
asw1-eqsin
set vlans XLink2 description "cr2-eqsin <-> mr1-eqsin"
set vlans XLink2 vlan-id 402
set interfaces interface-range infrastructure member xe-0/0/20
set interfaces interface-range infrastructure member xe-1/0/20
set interfaces xe-0/0/20 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-0/0/20 ether-options 802.3ad ae2
set interfaces xe-1/0/20 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-1/0/20 ether-options 802.3ad ae2
set interfaces ae2 description "Core: cr2-eqsin:ae1"
set interfaces ae2 mtu 9192
set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp periodic fast
set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 0 family ethernet-switching vlan members XLink2
set interfaces ae2 unit 0 family ethernet-switching vlan members private1-eqsin
set interfaces ae2 unit 0 family ethernet-switching vlan members public1-eqsin
set interfaces ae2 unit 0 family ethernet-switching vlan members sandbox1-eqsin

https://gerrit.wikimedia.org/r/c/operations/puppet/+/490525

cr1-eqsin
delete protocols bgp group PyBal neighbor 10.132.0.13
delete routing-options static route 103.102.166.224/28 next-hop 10.132.0.11
delete routing-options static route 103.102.166.240/28 next-hop 10.132.0.12
set routing-options static route 103.102.166.224/28 next-hop 10.132.0.13
set routing-options static route 103.102.166.240/28 next-hop 10.132.0.13
cr2-eqsin
set protocols bgp group PyBal neighbor 10.132.0.13
set routing-options static route 103.102.166.224/28 next-hop 10.132.0.11
set routing-options static route 103.102.166.240/28 next-hop 10.132.0.12
  • Verify working

Test redundancy

  • For each LACP bundles, unplug 1 member and verify no impact
  • Power down cr1-eqsin
  • Measure failover time, ensure acceptable
  • Ensure alerting works
  • Power back up
  • Power down cr2-eqsin
  • Measure failover time, ensure acceptable
  • Ensure alerting works
  • Power back up
  • Verify both routers are back to normal
  • Repool eqsin
  • Update Netbox (vlan, links, device) (remaining tackled in T205897)
  • Update interfaces descriptions
  • Update diagrams

Event Timeline

ayounsi triaged this task as Normal priority.Jan 7 2019, 10:57 PM
ayounsi created this task.
Restricted Application added a project: Operations. · View Herald TranscriptJan 7 2019, 10:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi claimed this task.Jan 25 2019, 8:34 PM
ayounsi added a project: ops-eqiad.
ayounsi updated the task description. (Show Details)
ayounsi reassigned this task from ayounsi to Cmjohnson.Jan 26 2019, 12:26 AM
ayounsi updated the task description. (Show Details)
In T205487#4932700, @Cmjohnson wrote:

@ayounsi This is the contents that I am shipping to eqsin. Please confirm that is all you need
12 SFP-10GLR Transceivers
(2) 3M LC-LC fiber
(4) 5M LC-LC fiber
(3) 5M LC-SC fiber
(1) mx204
(2) c13-c14 power cables

Confirmed (+mounting brackets or rails)
I'm pretty sure we got spares, but can you add 1 console cable, and 1 cat5 (mgmt), 1m for both? just in case.

RobH added a subscriber: RobH.Feb 11 2019, 4:35 PM

Chris shipped this, and I just put in an inbound shipemnt ticket for EQ Singapore SG#: 1-185487164544
UPS tracking 1Z291X71DG27842078

RobH updated the task description. (Show Details)Feb 11 2019, 4:35 PM

I also put in an in-bound ticket

1-185487164573

RobH added a comment.Feb 11 2019, 4:39 PM
This comment was removed by RobH.
RobH added a comment.Feb 11 2019, 4:47 PM

So deleting a ticket rquires us to open a 'delete request' ticket, seems easier to just keep both open and they'll receive the shipment in on one or the other.

RobH added a comment.Feb 12 2019, 7:24 PM

Chris shipped this, and I just put in an inbound shipemnt ticket for EQ Singapore SG#: 1-185487164544
UPS tracking 1Z291X71DG27842078

EQ SG3 detected the dupes, and cancelled 1-185487164544, leaving 1-185487164573 in place.

Change 490518 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Monitoring: add cr2-eqsin

https://gerrit.wikimedia.org/r/490518

ayounsi claimed this task.Feb 14 2019, 1:17 AM
ayounsi updated the task description. (Show Details)
ayounsi added a subscriber: Cmjohnson.
ayounsi updated the task description. (Show Details)Feb 14 2019, 3:42 AM

Change 490525 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Make lvs5003 peer with cr2-eqsin

https://gerrit.wikimedia.org/r/490525

ayounsi updated the task description. (Show Details)Feb 14 2019, 3:57 AM
ayounsi added subscribers: mark, faidon.

@BBlack @faidon @mark reviews welcome if you have some time!

Change 491889 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool eqsin for cr2-eqsin setup

https://gerrit.wikimedia.org/r/491889

Change 491889 merged by Ayounsi:
[operations/dns@master] Depool eqsin for cr2-eqsin setup

https://gerrit.wikimedia.org/r/491889

Mentioned in SAL (#wikimedia-operations) [2019-02-21T02:44:00Z] <XioNoX> depool eqsin - T213121

Change 491888 merged by Ayounsi:
[operations/dns@master] DNS: Add cr2-eqsin + related cr1 changes

https://gerrit.wikimedia.org/r/491888

Change 491894 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: add cr2-eqsin mgmt A

https://gerrit.wikimedia.org/r/491894

Change 491894 merged by Ayounsi:
[operations/dns@master] DNS: add cr2-eqsin mgmt A

https://gerrit.wikimedia.org/r/491894

ayounsi updated the task description. (Show Details)Feb 21 2019, 4:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T04:16:38Z] <XioNoX> Unplug Tata/NTT/PCCW from cr1-eqsin - T213121

ayounsi updated the task description. (Show Details)Feb 21 2019, 4:22 AM
ayounsi updated the task description. (Show Details)Feb 21 2019, 5:36 AM
ayounsi updated the task description. (Show Details)Feb 21 2019, 6:25 AM
ayounsi updated the task description. (Show Details)Feb 21 2019, 6:50 AM

Change 491903 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: cr2-eqsin A, cr11->cr2 renames where needed

https://gerrit.wikimedia.org/r/491903

Change 491903 merged by Ayounsi:
[operations/dns@master] DNS: cr2-eqsin A, cr1->cr2 renames where needed

https://gerrit.wikimedia.org/r/491903

Change 490518 merged by Ayounsi:
[operations/puppet@production] Monitoring: add cr2-eqsin

https://gerrit.wikimedia.org/r/490518

ayounsi updated the task description. (Show Details)Feb 21 2019, 7:24 AM

Change 491925 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Icinga: add cr2-eqsin mgmt interface

https://gerrit.wikimedia.org/r/491925

Change 491925 merged by Ayounsi:
[operations/puppet@production] Icinga: add cr2-eqsin mgmt interface

https://gerrit.wikimedia.org/r/491925

Change 491964 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Revert "Depool eqsin for cr2-eqsin setup"

https://gerrit.wikimedia.org/r/491964

Change 491964 merged by BBlack:
[operations/dns@master] Revert "Depool eqsin for cr2-eqsin setup"

https://gerrit.wikimedia.org/r/491964

Mentioned in SAL (#wikimedia-operations) [2019-02-22T03:26:19Z] <XioNoX> delete old gr-1/0/0 from cr1-eqsin - T213121

ayounsi updated the task description. (Show Details)Feb 22 2019, 4:24 AM
ayounsi updated the task description. (Show Details)Feb 22 2019, 4:33 AM

Change 490525 merged by Vgutierrez:
[operations/puppet@production] Make lvs5003 peer with cr2-eqsin

https://gerrit.wikimedia.org/r/490525

Mentioned in SAL (#wikimedia-operations) [2019-02-25T19:41:51Z] <vgutierrez> restarting pybal on lvs5003 - T213121

ayounsi updated the task description. (Show Details)Feb 25 2019, 7:51 PM

Change 494278 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] pybal: switch lvs5002 BGP peering from cr1-eqsin to cr2-eqsin

https://gerrit.wikimedia.org/r/494278

Moving lvs5002 to cr2-eqsin
1/ push the following (no impact)

cr2-eqsin
[edit routing-options rib inet6.0 static route 2001:df2:e500:ed1a::2:0/111]
-     next-hop 2001:df2:e500:101:10:132:0:12;
+     next-hop 2001:df2:e500:101:10:132:0:11;
[edit routing-options static route 103.102.166.240/28]
-    next-hop 10.132.0.12;
+    next-hop 10.132.0.11;
[edit protocols bgp group PyBal]
      neighbor 10.132.0.13 { ... }
+     neighbor 10.132.0.12;

2/ Disable pybal
3/ Merge https://gerrit.wikimedia.org/r/494278
4/ Ensure BGP is up with the good peer
5/ push

cr1-eqsin
[edit protocols bgp group PyBal]
-     neighbor 10.132.0.12;

Mentioned in SAL (#wikimedia-operations) [2019-03-04T18:16:38Z] <XioNoX> push lvs5002 changes on cr2-eqsin - T213121

Change 494278 merged by Vgutierrez:
[operations/puppet@production] pybal: switch lvs5002 BGP peering from cr1-eqsin to cr2-eqsin

https://gerrit.wikimedia.org/r/494278

Mentioned in SAL (#wikimedia-operations) [2019-03-04T18:23:52Z] <vgutierrez> restarting pybal on lvs5002 - T213121

ayounsi closed this task as Resolved.Mar 8 2019, 5:20 PM

the redundancy testing is outside the scope of the goal, so everything needed here is done.

faidon mentioned this in Unknown Object (Task).Apr 19 2019, 11:29 AM