Page MenuHomePhabricator
Create Task
Maniphest T213121

Deploy cr2-eqsin
Closed, ResolvedPublic
Actions

Description

  • Figure out who is going to do the onsite work (Arzhel)
  • Decide on a date Feb. 21st/22nd
  • Verify router config is up to date with standards [Arzhel]
  • Ship router and associated cables/optics to arrive on Feb 20th latest

Onsite

  • Depool eqsin
  • Rack cr2-eqsin in U44
  • Connect mgmt/console/power
  • Verify working
  • Unplug Tata/NTT/PCCW from cr1-eqsin
  • Delete BGP config
  • Reconfigure ports:

cr1:xe-1/0/0 to cr2:xe-0/1/5
cr1:xe-2/0/0 to cr2:xe-0/1/6

cr1-eqsin
delete interfaces xe-1/0/0
delete interfaces xe-2/0/0
delete interfaces xe-2/0/2
show interfaces interface-range external member xe-1/0/0
show interfaces interface-range external member xe-2/0/0
show interfaces interface-range external member xe-2/0/2
set interfaces interface-range disabled member xe-2/0/2
set interfaces interface-range disabled description DISABLED
set interfaces interface-range disabled disable
set interfaces xe-1/0/0 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-1/0/0 gigether-options 802.3ad ae0
set interfaces xe-2/0/0 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-2/0/0 gigether-options 802.3ad ae0
set interfaces ae0 description "Core: cr2-eqsin:ae0"
set interfaces ae0 mtu 9192
set interfaces ae0 aggregated-ether-options link-speed mixed
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp periodic fast
set interfaces ae0 unit 0 family inet address 103.102.166.140/31
set interfaces ae0 unit 0 family inet6 address 2001:df2:e500:fe05::1/64
set protocols ospf area 0.0.0.0 interface ae0.0 interface-type p2p
set protocols ospf3 area 0.0.0.0 interface ae0.0 interface-type p2p
set protocols pim interface ae0.0
set protocols bgp group Confed_eqsin type internal
set protocols bgp group Confed_eqsin local-address 103.102.166.129
set protocols bgp group Confed_eqsin family inet any
set protocols bgp group Confed_eqsin family inet6 any
set protocols bgp group Confed_eqsin peer-as 65005
set protocols bgp group Confed_eqsin local-as 65005
set protocols bgp group Confed_eqsin local-as no-prepend-global-as
set protocols bgp group Confed_eqsin neighbor 103.102.166.130
  • Connect cr1<-->cr2
  • Verify link/OSPF/BGP up
  • Connect cr2:xe-0/1/3 (already configured)<-->asw:xe-0/0/20
  • Connect cr2:xe-0/1/4 (already configured)<-->asw:xe-1/0/20
cr1-eqsin
delete interfaces ae1 unit 510 family inet address 103.102.166.1/28
delete interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1::1/64
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 virtual-address 103.102.166.1
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 accept-data
set interfaces ae1 unit 510 family inet address 103.102.166.2/28 vrrp-group 1 track interface ae1.510 priority-cost 50
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:1::1
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 accept-data
set interfaces ae1 unit 510 family inet6 address 2001:df2:e500:1:fe00::1/64 vrrp-inet6-group 1 track interface ae1.510 priority-cost 50
set interfaces ae1 unit 510 family inet6 address fe80:0:0:0:fe00::1/64

delete interfaces ae1 unit 520 family inet address 10.132.0.1/24
delete interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101::1/64
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 virtual-address 10.132.0.1
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 accept-data
set interfaces ae1 unit 520 family inet address 10.132.0.2/24 vrrp-group 1 track interface ae1.520 priority-cost 50
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:101::1
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 accept-data
set interfaces ae1 unit 520 family inet6 address 2001:df2:e500:101:fe00::1/64 vrrp-inet6-group 1 track interface ae1.520 priority-cost 50
set interfaces ae1 unit 520 family inet6 address fe80:0:0:0:fe00::1/64

delete interfaces ae1 unit 530 family inet address 103.102.166.17/28
delete interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201::1/64
set interfaces ae1 unit 530 family inet address 103.102.166.18/28 vrrp-group 1 virtual-address 103.102.166.17
set interfaces ae1 unit 530 family inet address 103.102.166.18/28 vrrp-group 1 track interface ae1.530 priority-cost 50
set interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201:fe00::1/64 vrrp-inet6-group 1 virtual-inet6-address 2001:df2:e500:201::1
set interfaces ae1 unit 530 family inet6 address 2001:df2:e500:201:fe00::1/64 vrrp-inet6-group 1 track interface ae1.530 priority-cost 50
set interfaces ae1 unit 530 family inet6 address fe80:0:0:0:fe00::1/64
asw1-eqsin
set vlans XLink2 description "cr2-eqsin <-> mr1-eqsin"
set vlans XLink2 vlan-id 402
set interfaces interface-range infrastructure member xe-0/0/20
set interfaces interface-range infrastructure member xe-1/0/20
set interfaces xe-0/0/20 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-0/0/20 ether-options 802.3ad ae2
set interfaces xe-1/0/20 description "Core: cr2-eqsin:??? {#???} [10Gbps DF]"
set interfaces xe-1/0/20 ether-options 802.3ad ae2
set interfaces ae2 description "Core: cr2-eqsin:ae1"
set interfaces ae2 mtu 9192
set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp periodic fast
set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 0 family ethernet-switching vlan members XLink2
set interfaces ae2 unit 0 family ethernet-switching vlan members private1-eqsin
set interfaces ae2 unit 0 family ethernet-switching vlan members public1-eqsin
set interfaces ae2 unit 0 family ethernet-switching vlan members sandbox1-eqsin

https://gerrit.wikimedia.org/r/c/operations/puppet/+/490525

cr1-eqsin
delete protocols bgp group PyBal neighbor 10.132.0.13
delete routing-options static route 103.102.166.224/28 next-hop 10.132.0.11
delete routing-options static route 103.102.166.240/28 next-hop 10.132.0.12
set routing-options static route 103.102.166.224/28 next-hop 10.132.0.13
set routing-options static route 103.102.166.240/28 next-hop 10.132.0.13
cr2-eqsin
set protocols bgp group PyBal neighbor 10.132.0.13
set routing-options static route 103.102.166.224/28 next-hop 10.132.0.11
set routing-options static route 103.102.166.240/28 next-hop 10.132.0.12
  • Verify working

Test redundancy

  • For each LACP bundles, unplug 1 member and verify no impact
  • Power down cr1-eqsin
  • Measure failover time, ensure acceptable
  • Ensure alerting works
  • Power back up
  • Power down cr2-eqsin
  • Measure failover time, ensure acceptable
  • Ensure alerting works
  • Power back up
  • Verify both routers are back to normal
  • Repool eqsin
  • Update Netbox (vlan, links, device) (remaining tackled in T205897)
  • Update interfaces descriptions
  • Update diagrams

Details

SubjectRepoBranchLines +/-
pybal: switch lvs5002 BGP peering from cr1-eqsin to cr2-eqsinoperations/puppetproduction+2 -2
Make lvs5003 peer with cr2-eqsinoperations/puppetproduction+2 -1
Revert "Depool eqsin for cr2-eqsin setup"operations/dnsmaster+0 -3
Icinga: add cr2-eqsin mgmt interfaceoperations/puppetproduction+2 -2
Monitoring: add cr2-eqsinoperations/puppetproduction+10 -1
DNS: cr2-eqsin A, cr1->cr2 renames where neededoperations/dnsmaster+9 -7
DNS: add cr2-eqsin mgmt Aoperations/dnsmaster+1 -0
DNS: Add cr2-eqsin + related cr1 changesoperations/dnsmaster+41 -7
Depool eqsin for cr2-eqsin setupoperations/dnsmaster+3 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• ayounsiT213122 Increase network capacity (2018-19 Q3 Goal)
 Resolved• ayounsiT213121 Deploy cr2-eqsin

Event Timeline

• ayounsi triaged this task as Medium priority.Jan 7 2019, 10:57 PM
• ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptJan 7 2019, 10:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ayounsi mentioned this in T213122: Increase network capacity (2018-19 Q3 Goal).Jan 7 2019, 10:59 PM
• ayounsi added a parent task: T213122: Increase network capacity (2018-19 Q3 Goal).
Paladox subscribed.Jan 8 2019, 12:15 AM
• ayounsi claimed this task.Jan 25 2019, 8:34 PM
• ayounsi added a project: ops-eqiad.
• ayounsi updated the task description. (Show Details)
• ayounsi reassigned this task from • ayounsi to • Cmjohnson.Jan 26 2019, 12:26 AM
• ayounsi updated the task description. (Show Details)
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 11:00 PM
• ayounsi added a comment.Feb 6 2019, 7:22 PM
In T205487#4932700, @Cmjohnson wrote:

@ayounsi This is the contents that I am shipping to eqsin. Please confirm that is all you need

12 SFP-10GLR Transceivers
(2) 3M LC-LC fiber
(4) 5M LC-LC fiber
(3) 5M LC-SC fiber

(1) mx204
(2) c13-c14 power cables

Confirmed (+mounting brackets or rails)
I'm pretty sure we got spares, but can you add 1 console cable, and 1 cat5 (mgmt), 1m for both? just in case.

RobH subscribed.Feb 11 2019, 4:35 PM

Chris shipped this, and I just put in an inbound shipemnt ticket for EQ Singapore SG#: 1-185487164544
UPS tracking 1Z291X71DG27842078

RobH updated the task description. (Show Details)Feb 11 2019, 4:35 PM
• Cmjohnson added a comment.Feb 11 2019, 4:38 PM

I also put in an in-bound ticket

1-185487164573

RobH added a comment.Feb 11 2019, 4:39 PM
This comment was removed by RobH.
RobH added a comment.Feb 11 2019, 4:47 PM

So deleting a ticket rquires us to open a 'delete request' ticket, seems easier to just keep both open and they'll receive the shipment in on one or the other.

RobH added a comment.Feb 12 2019, 7:24 PM
In T213121#4944125, @RobH wrote:

Chris shipped this, and I just put in an inbound shipemnt ticket for EQ Singapore SG#: 1-185487164544
UPS tracking 1Z291X71DG27842078

EQ SG3 detected the dupes, and cancelled 1-185487164544, leaving 1-185487164573 in place.

gerritbot subscribed.Feb 14 2019, 1:16 AM

Change 490518 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Monitoring: add cr2-eqsin

https://gerrit.wikimedia.org/r/490518

gerritbot added a project: Patch-For-Review.Feb 14 2019, 1:16 AM
• ayounsi claimed this task.Feb 14 2019, 1:17 AM
• ayounsi updated the task description. (Show Details)
• ayounsi added a subscriber: • Cmjohnson.
• ayounsi updated the task description. (Show Details)Feb 14 2019, 3:42 AM
gerritbot added a comment.Feb 14 2019, 3:55 AM

Change 490525 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Make lvs5003 peer with cr2-eqsin

https://gerrit.wikimedia.org/r/490525

• ayounsi updated the task description. (Show Details)Feb 14 2019, 3:57 AM
• ayounsi added subscribers: mark, faidon.

@BBlack @faidon @mark reviews welcome if you have some time!

gerritbot added a comment.Feb 21 2019, 2:34 AM

Change 491889 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Depool eqsin for cr2-eqsin setup

https://gerrit.wikimedia.org/r/491889

gerritbot added a comment.Feb 21 2019, 2:43 AM

Change 491889 merged by Ayounsi:
[operations/dns@master] Depool eqsin for cr2-eqsin setup

https://gerrit.wikimedia.org/r/491889

Stashbot subscribed.Feb 21 2019, 2:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T02:44:00Z] <XioNoX> depool eqsin - T213121

gerritbot added a comment.Feb 21 2019, 3:52 AM

Change 491888 merged by Ayounsi:
[operations/dns@master] DNS: Add cr2-eqsin + related cr1 changes

https://gerrit.wikimedia.org/r/491888

gerritbot added a comment.Feb 21 2019, 4:06 AM

Change 491894 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: add cr2-eqsin mgmt A

https://gerrit.wikimedia.org/r/491894

gerritbot added a comment.Feb 21 2019, 4:08 AM

Change 491894 merged by Ayounsi:
[operations/dns@master] DNS: add cr2-eqsin mgmt A

https://gerrit.wikimedia.org/r/491894

• ayounsi updated the task description. (Show Details)Feb 21 2019, 4:16 AM
Stashbot added a comment.Feb 21 2019, 4:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T04:16:38Z] <XioNoX> Unplug Tata/NTT/PCCW from cr1-eqsin - T213121

• ayounsi updated the task description. (Show Details)Feb 21 2019, 4:22 AM
• ayounsi updated the task description. (Show Details)Feb 21 2019, 5:36 AM
• ayounsi updated the task description. (Show Details)Feb 21 2019, 6:25 AM
• ayounsi updated the task description. (Show Details)Feb 21 2019, 6:50 AM
gerritbot added a comment.Feb 21 2019, 6:58 AM

Change 491903 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: cr2-eqsin A, cr11->cr2 renames where needed

https://gerrit.wikimedia.org/r/491903

gerritbot added a comment.Feb 21 2019, 7:00 AM

Change 491903 merged by Ayounsi:
[operations/dns@master] DNS: cr2-eqsin A, cr1->cr2 renames where needed

https://gerrit.wikimedia.org/r/491903

gerritbot added a comment.Feb 21 2019, 7:05 AM

Change 490518 merged by Ayounsi:
[operations/puppet@production] Monitoring: add cr2-eqsin

https://gerrit.wikimedia.org/r/490518

• ayounsi updated the task description. (Show Details)Feb 21 2019, 7:24 AM
• ayounsi added a comment.Feb 21 2019, 9:47 AM

Also audited all the spares we have onsite:
https://docs.google.com/spreadsheets/d/1FKYVQJePjTQ7nVwYv4oDC6Gszk7RLrkq5ySN0fjvSoY/edit#gid=2057953856
Labelled the few cables not labelled:
https://docs.google.com/spreadsheets/d/1FKYVQJePjTQ7nVwYv4oDC6Gszk7RLrkq5ySN0fjvSoY/edit#gid=1665726692

gerritbot added a comment.Feb 21 2019, 10:06 AM

Change 491925 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Icinga: add cr2-eqsin mgmt interface

https://gerrit.wikimedia.org/r/491925

gerritbot added a comment.Feb 21 2019, 10:10 AM

Change 491925 merged by Ayounsi:
[operations/puppet@production] Icinga: add cr2-eqsin mgmt interface

https://gerrit.wikimedia.org/r/491925

gerritbot added a comment.Feb 21 2019, 2:04 PM

Change 491964 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Revert "Depool eqsin for cr2-eqsin setup"

https://gerrit.wikimedia.org/r/491964

gerritbot added a comment.Feb 21 2019, 7:58 PM

Change 491964 merged by BBlack:
[operations/dns@master] Revert "Depool eqsin for cr2-eqsin setup"

https://gerrit.wikimedia.org/r/491964

Stashbot added a comment.Feb 22 2019, 3:26 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-22T03:26:19Z] <XioNoX> delete old gr-1/0/0 from cr1-eqsin - T213121

• ayounsi updated the task description. (Show Details)Feb 22 2019, 4:24 AM
• ayounsi updated the task description. (Show Details)Feb 22 2019, 4:33 AM
gerritbot added a comment.Feb 25 2019, 7:38 PM

Change 490525 merged by Vgutierrez:
[operations/puppet@production] Make lvs5003 peer with cr2-eqsin

https://gerrit.wikimedia.org/r/490525

Stashbot added a comment.Feb 25 2019, 7:41 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-25T19:41:51Z] <vgutierrez> restarting pybal on lvs5003 - T213121

• ayounsi updated the task description. (Show Details)Feb 25 2019, 7:51 PM
gerritbot added a comment.Mar 4 2019, 5:58 PM

Change 494278 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] pybal: switch lvs5002 BGP peering from cr1-eqsin to cr2-eqsin

https://gerrit.wikimedia.org/r/494278

• ayounsi added a comment.Mar 4 2019, 6:13 PM

Moving lvs5002 to cr2-eqsin
1/ push the following (no impact)

cr2-eqsin
[edit routing-options rib inet6.0 static route 2001:df2:e500:ed1a::2:0/111]
-     next-hop 2001:df2:e500:101:10:132:0:12;
+     next-hop 2001:df2:e500:101:10:132:0:11;
[edit routing-options static route 103.102.166.240/28]
-    next-hop 10.132.0.12;
+    next-hop 10.132.0.11;
[edit protocols bgp group PyBal]
      neighbor 10.132.0.13 { ... }
+     neighbor 10.132.0.12;

2/ Disable pybal
3/ Merge https://gerrit.wikimedia.org/r/494278
4/ Ensure BGP is up with the good peer
5/ push

cr1-eqsin
[edit protocols bgp group PyBal]
-     neighbor 10.132.0.12;
Stashbot added a comment.Mar 4 2019, 6:16 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-04T18:16:38Z] <XioNoX> push lvs5002 changes on cr2-eqsin - T213121

gerritbot added a comment.Mar 4 2019, 6:19 PM

Change 494278 merged by Vgutierrez:
[operations/puppet@production] pybal: switch lvs5002 BGP peering from cr1-eqsin to cr2-eqsin

https://gerrit.wikimedia.org/r/494278

Stashbot added a comment.Mar 4 2019, 6:23 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-04T18:23:52Z] <vgutierrez> restarting pybal on lvs5002 - T213121

• ayounsi closed this task as Resolved.Mar 8 2019, 5:20 PM

the redundancy testing is outside the scope of the goal, so everything needed here is done.

faidon mentioned this in Unknown Object (Task).Apr 19 2019, 11:29 AM
faidon mentioned this in rOHPU17e59ece8e86: Add cr2-eqsin.Oct 3 2019, 11:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits