Page MenuHomePhabricator

Rack/setup cr2-eqdfw
Closed, ResolvedPublic

Description

The router is racked for now in C5 and connected to scs-c1-codfw port 34

Before moving cr2-eqdfw:

  • Disable external BGP sessions on cr2 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Enable remote root login [Arzhel]
  • Notify GTT of the MAC address change [Arzhel]
  • Temporarily duplicate cr1:xe-0/0/3 MAC on cr2:xe-0/1/3 [Arzhel]

In eqdfw - August 23rd, 11am CDT, 9am PDT, 4pm UTC. 1h.

  • Rack and power on cr2 [Papaul]
  • Downtime monitoring [Arzhel]
  • Disable external BGP sessions on cr1 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Ensure no more traffic is flowing through cr1 [Arzhel]
  • Power down cr1 [Arzhel]
  • Move xe-0/0/0 -> xe-0/1/0 [Papaul]
  • Ensure device is reachable remotely [Arzhel]
  • Move remaining links [Papaul]

xe-0/0/1 -> xe-0/1/1
xe-0/0/2 -> xe-0/1/2
xe-0/0/3 -> xe-0/1/3
xe-1/0/0 -> xe-0/1/4
xe-1/1/0 -> xe-0/1/5
xe-1/2/0 -> xe-0/1/6
xe-1/3/0 -> xe-0/1/7

  • Ensure device is healthy [Arzhel]
  • Set root password to standard + disable root login [Arzhel]
  • Enable external BGP sessions [Arzhel]

If issues:

  • Connect to serial with temp root password + screen sharing [Papaul]
  • Connect to device using LL IP from neighbor router [Arzhel]
  • Rollback to previous router [Papaul]

After move:

  • Update interfaces descriptions [Arzhel]
  • Update DNS [Arzhel]
  • Update Smokeping [Arzhel]
  • Update racktables [Papaul]
  • Update monitoring (LibreNMS+bills/Icinga) [Arzhel]
  • Remove GTT MAC filtering workaround [Arzhel]

Long after:

Related Objects

StatusSubtypeAssignedTask
Invalidayounsi
Resolvedayounsi

Event Timeline

Papaul triaged this task as Medium priority.Jun 11 2018, 7:42 PM
Papaul created this task.

As the current MX80 uses XFP-10G-LR optics and the MX204 uses EX-SFP-10GE-LR we're going to need 5*EX-SFP-10GE-LR optics (+ at least one spare).

@Papaul how many EX-SFP-10GE-LR optics do you have? I'll open a procurement task if not enough.

Thanks.

@ayounsi I have none. I had 12 left but I used them to connect the lvs2009 and lvs2010

ayounsi mentioned this in Unknown Object (Task).Jun 25 2018, 5:35 PM
Vvjjkkii renamed this task from Rack/setup cr2-eqdfw to q8aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii removed ayounsi as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.

@Papaul, can you pre-populate the following interfaces with SFP+-10G-LR?

xe-0/1/0   {#11399}
xe-0/1/1   {#11401} 
xe-0/1/3   {#11389}
xe-0/1/4   {#11403}
xe-0/1/6   {#11397}

You can store a few spares in xe-0/1/2, xe-0/1/5, xe-0/1/7 as well.

Old/new router cable matching is the following:
xe-0/0/0 -> xe-0/1/0
xe-0/0/1 -> xe-0/1/1
xe-0/0/2 -> xe-0/1/2
xe-0/0/3 -> xe-0/1/3
xe-1/0/0 -> xe-0/1/4
xe-1/1/0 -> xe-0/1/5
xe-1/2/0 -> xe-0/1/6
xe-1/3/0 -> xe-0/1/7

Planning on doing the swap on August 14th, 11am CDT, 9am PDT, 4pm UTC. 1h.

Pending no planned maintenance from redundant links.

EDIT: August 23rd, 11am CDT, 9am PDT, 4pm UTC. 1h.

Papaul closed subtask Unknown Object (Task) as Resolved.Aug 6 2018, 9:44 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T16:17:09Z] <XioNoX> deactivating IX/Transit links on cr1-eqdfw - T196941

Mentioned in SAL (#wikimedia-operations) [2018-08-23T17:16:41Z] <XioNoX> enable v6 neighbors on cr2-eqdfw - T196941

Mentioned in SAL (#wikimedia-operations) [2018-08-23T17:28:22Z] <XioNoX> enable v4 neighbors on cr2-eqdfw - T196941

Change 454865 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Puppet: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454865

Change 454867 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454867

Change 454867 merged by Ayounsi:
[operations/dns@master] DNS: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454867

Change 454865 merged by Ayounsi:
[operations/puppet@production] Puppet: rename cr1-eqdfw to cr2-eqdfw

https://gerrit.wikimedia.org/r/454865

Mentioned in SAL (#wikimedia-operations) [2018-08-23T18:30:04Z] <XioNoX> remove MAC filter workaround on cr2-eqdfw - T196941

Mentioned in SAL (#wikimedia-operations) [2018-08-23T18:33:31Z] <XioNoX> Equinix updated their MAC filter, IX sessions up - T196941

ayounsi updated the task description. (Show Details)