Page MenuHomePhabricator
Create Task
Maniphest T196941

Rack/setup cr2-eqdfw
Closed, ResolvedPublic
Actions

Description

The router is racked for now in C5 and connected to scs-c1-codfw port 34

Before moving cr2-eqdfw:

  • Disable external BGP sessions on cr2 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Enable remote root login [Arzhel]
  • Notify GTT of the MAC address change [Arzhel]
  • Temporarily duplicate cr1:xe-0/0/3 MAC on cr2:xe-0/1/3 [Arzhel]

In eqdfw - August 23rd, 11am CDT, 9am PDT, 4pm UTC. 1h.

  • Rack and power on cr2 [Papaul]
  • Downtime monitoring [Arzhel]
  • Disable external BGP sessions on cr1 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Ensure no more traffic is flowing through cr1 [Arzhel]
  • Power down cr1 [Arzhel]
  • Move xe-0/0/0 -> xe-0/1/0 [Papaul]
  • Ensure device is reachable remotely [Arzhel]
  • Move remaining links [Papaul]

xe-0/0/1 -> xe-0/1/1
xe-0/0/2 -> xe-0/1/2
xe-0/0/3 -> xe-0/1/3
xe-1/0/0 -> xe-0/1/4
xe-1/1/0 -> xe-0/1/5
xe-1/2/0 -> xe-0/1/6
xe-1/3/0 -> xe-0/1/7

  • Ensure device is healthy [Arzhel]
  • Set root password to standard + disable root login [Arzhel]
  • Enable external BGP sessions [Arzhel]

If issues:

  • Connect to serial with temp root password + screen sharing [Papaul]
  • Connect to device using LL IP from neighbor router [Arzhel]
  • Rollback to previous router [Papaul]

After move:

  • Update interfaces descriptions [Arzhel]
  • Update DNS [Arzhel]
  • Update Smokeping [Arzhel]
  • Update racktables [Papaul]
  • Update monitoring (LibreNMS+bills/Icinga) [Arzhel]
  • Remove GTT MAC filtering workaround [Arzhel]

Long after:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Puppet: rename cr1-eqdfw to cr2-eqdfwoperations/puppetproduction+6 -6
DNS: rename cr1-eqdfw to cr2-eqdfwoperations/dnsmaster+22 -22
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 InvalidayounsiT199142 Increase network capacity (2018-19 Q1 Goal)
 ResolvedayounsiT196941 Rack/setup cr2-eqdfw
Unknown Object (Task)

Event Timeline

Papaul created this task.Jun 11 2018, 7:42 PM
Papaul triaged this task as Medium priority.
ayounsi added a comment.Jun 25 2018, 2:08 PM

As the current MX80 uses XFP-10G-LR optics and the MX204 uses EX-SFP-10GE-LR we're going to need 5*EX-SFP-10GE-LR optics (+ at least one spare).

@Papaul how many EX-SFP-10GE-LR optics do you have? I'll open a procurement task if not enough.

Thanks.

Papaul added a comment.Jun 25 2018, 4:47 PM

@ayounsi I have none. I had 12 left but I used them to connect the lvs2009 and lvs2010

ayounsi mentioned this in Unknown Object (Task).Jun 25 2018, 5:35 PM
• Vvjjkkii renamed this task from Rack/setup cr2-eqdfw to q8aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii removed ayounsi as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from q8aaaaaaaa to Rack/setup cr2-eqdfw.Jul 2 2018, 9:25 AM
CommunityTechBot assigned this task to ayounsi.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
ayounsi added a parent task: T199142: Increase network capacity (2018-19 Q1 Goal).Jul 9 2018, 6:31 PM
ayounsi added a subtask: Unknown Object (Task).
ayounsi added a comment.Aug 6 2018, 9:25 PM

@Papaul, can you pre-populate the following interfaces with SFP+-10G-LR?

xe-0/1/0   {#11399}
xe-0/1/1   {#11401} 
xe-0/1/3   {#11389}
xe-0/1/4   {#11403}
xe-0/1/6   {#11397}

You can store a few spares in xe-0/1/2, xe-0/1/5, xe-0/1/7 as well.

Old/new router cable matching is the following:
xe-0/0/0 -> xe-0/1/0
xe-0/0/1 -> xe-0/1/1
xe-0/0/2 -> xe-0/1/2
xe-0/0/3 -> xe-0/1/3
xe-1/0/0 -> xe-0/1/4
xe-1/1/0 -> xe-0/1/5
xe-1/2/0 -> xe-0/1/6
xe-1/3/0 -> xe-0/1/7

ayounsi added a comment.EditedAug 6 2018, 9:37 PM

Planning on doing the swap on August 14th, 11am CDT, 9am PDT, 4pm UTC. 1h.

Pending no planned maintenance from redundant links.

EDIT: August 23rd, 11am CDT, 9am PDT, 4pm UTC. 1h.

Papaul closed subtask Unknown Object (Task) as Resolved.Aug 6 2018, 9:44 PM
Papaul added a comment.Aug 8 2018, 2:57 PM

@ayounsi all SFP+-10G-LR are in place .

ayounsi added a project: ops-eqdfw.Aug 21 2018, 5:18 PM
ayounsi updated the task description. (Show Details)
ayounsi updated the task description. (Show Details)Aug 21 2018, 7:38 PM
Stashbot subscribed.Aug 23 2018, 4:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T16:17:09Z] <XioNoX> deactivating IX/Transit links on cr1-eqdfw - T196941

ayounsi updated the task description. (Show Details)Aug 23 2018, 4:26 PM
Stashbot added a comment.Aug 23 2018, 5:01 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T17:01:52Z] <XioNoX> powering off cr1-eqdfw - T196941

Papaul updated the task description. (Show Details)Aug 23 2018, 5:02 PM
ayounsi updated the task description. (Show Details)Aug 23 2018, 5:15 PM
Stashbot added a comment.Aug 23 2018, 5:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T17:16:41Z] <XioNoX> enable v6 neighbors on cr2-eqdfw - T196941

Stashbot added a comment.Aug 23 2018, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T17:28:22Z] <XioNoX> enable v4 neighbors on cr2-eqdfw - T196941

ayounsi updated the task description. (Show Details)Aug 23 2018, 5:43 PM
gerritbot subscribed.Aug 23 2018, 6:00 PM

Change 454865 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Puppet: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454865

gerritbot added a project: Patch-For-Review.Aug 23 2018, 6:00 PM
gerritbot added a comment.Aug 23 2018, 6:04 PM

Change 454867 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454867

gerritbot added a comment.Aug 23 2018, 6:05 PM

Change 454867 merged by Ayounsi:
[operations/dns@master] DNS: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454867

gerritbot added a comment.Aug 23 2018, 6:05 PM

Change 454865 merged by Ayounsi:
[operations/puppet@production] Puppet: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454865

Stashbot added a comment.Aug 23 2018, 6:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T18:30:04Z] <XioNoX> remove MAC filter workaround on cr2-eqdfw - T196941

ayounsi updated the task description. (Show Details)Aug 23 2018, 6:30 PM
Stashbot added a comment.Aug 23 2018, 6:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T18:33:31Z] <XioNoX> Equinix updated their MAC filter, IX sessions up - T196941

Papaul updated the task description. (Show Details)Aug 23 2018, 9:22 PM
ayounsi closed this task as Resolved.Aug 23 2018, 11:48 PM
ayounsi updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits