Page MenuHomePhabricator

Rack/setup cr2-eqord
Closed, ResolvedPublic

Description

In eqiad:

  • Rack and connect power/serial [Chris]
  • Pre-populate all 8*10G ports with SFP+-10G-LR [Chris]
  • Connect USB drive with provided image [Chris]
  • Configure/upgrade router [Arzhel]
  • Enable remote root login + temp password [Arzhel]
  • Disable external BGP sessions on cr2 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Update config if changed + power down [Arzhel]
  • Ship device to eqord [Chris]

In eqord - Oct. 24rd 11:00am (Chicago time) 1h.

  • Rack and power-on cr2 [Papaul]
  • Ensure no ongoing provider maintenance [Arzhel]
  • Downtime monitoring [Arzhel]
  • Disable external BGP sessions on cr1 [Arzhel]
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
  • Ensure no more traffic is flowing through cr1 [Arzhel]
  • Power down cr1 [Arzhel]
  • Move xe-0/0/0 -> xe-0/1/0 transport link [Papaul]
  • Ensure device is reachable remotely [Arzhel]
  • Move remaining links [Papaul]
xe-0/0/1 -> xe-0/1/1
xe-0/0/2 -> xe-0/1/2
xe-0/0/3 -> xe-0/1/3
xe-1/0/0 -> xe-0/1/4
xe-1/1/0 -> xe-0/1/5
xe-1/2/0 -> xe-0/1/6
xe-1/3/0 -> xe-0/1/7
  • Ensure device is healthy [Arzhel]
  • Set root password to standard + disable root login [Arzhel]
  • Enable external BGP sessions [Arzhel]

If issues:

  • Connect to serial with temp root password + screen sharing [Papaul]
  • Connect to device using LL IP from neighbor router [Arzhel]
  • Rollback to previous router [Papaul]

After move:

  • Update interfaces descriptions [Arzhel]
  • Update DNS [Arzhel]
  • Update Smokeping [Arzhel]
  • Update Netbox [Papaul]
  • Update monitoring (LibreNMS+bills/Icinga) [Arzhel]
  • Open task for cr1 decom [Papaul] T208049

Event Timeline

ayounsi triaged this task as Medium priority.Sep 12 2018, 10:24 PM
ayounsi created this task.
Restricted Application added a project: Operations. · View Herald TranscriptSep 12 2018, 10:24 PM
Cmjohnson updated the task description. (Show Details)Sep 17 2018, 3:49 PM

@ayounsi I connected both mx204's I have in eqiad to the console and mgmt switch. cr2-eqord is on port 47 and the other is in port 48 and labled cr2-eqsin.

Cmjohnson moved this task from Backlog to Blocked on the ops-eqiad board.Sep 17 2018, 3:52 PM
ayounsi updated the task description. (Show Details)Sep 17 2018, 7:40 PM
ayounsi updated the task description. (Show Details)Sep 17 2018, 8:30 PM
ayounsi updated the task description. (Show Details)Sep 20 2018, 5:14 PM
ayounsi updated the task description. (Show Details)Oct 17 2018, 11:10 AM

@Cmjohnson all yours to populate with optics and ship to eqord, @Papaul will be there on Oct. 23rd.
If we don't have enough optics, please use the ones received for cr2-eqsin in T205487 and let me know.

RobH added a subscriber: RobH.Oct 19 2018, 4:23 PM

So I just gave @Papaul full rights to access CH2, identical to his rights in DA1 (so he should have no issues with access.)

Papaul updated the task description. (Show Details)Oct 24 2018, 12:25 AM

Went on site today to do my hand scan, get my access code and have an idea where things are . Got the router from shipping and racked it already.

@chris can you please update cr2-eqord Custom Fields in netbox when I am on site tomorrow I will put in the asset tag information. Or if you have the purchase task number you can just ink it to this task and i can look at the packing slip to do it. Thanks

@ayounsi What time do you want to start working on this today? I can be on site by 10:30 am Chicago time.

11:00 Chicago time works for me, I sent you a calendar invitation.

ayounsi updated the task description. (Show Details)Oct 24 2018, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-24T15:59:50Z] <XioNoX> disable BGP sessions to transit/peering on cr1-eqord - T204170

Mentioned in SAL (#wikimedia-operations) [2018-10-24T16:04:02Z] <XioNoX> power-off cr1-eqord - T204170

Change 469474 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Puppet: rename cr1-eqord to cr2-eqord

https://gerrit.wikimedia.org/r/469474

Change 469476 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] DNS: rename cr1-eqord to cr2-eqord

https://gerrit.wikimedia.org/r/469476

Change 469474 merged by Ayounsi:
[operations/puppet@production] Puppet: rename cr1-eqord to cr2-eqord

https://gerrit.wikimedia.org/r/469474

Change 469476 merged by Ayounsi:
[operations/dns@master] DNS: rename cr1-eqord to cr2-eqord

https://gerrit.wikimedia.org/r/469476

Mentioned in SAL (#wikimedia-operations) [2018-10-24T18:16:43Z] <XioNoX> enable BGP sessions to transit/peering on cr2-eqord - T204170

ayounsi updated the task description. (Show Details)Oct 24 2018, 6:31 PM
ayounsi reassigned this task from Cmjohnson to Papaul.Oct 24 2018, 6:43 PM
ayounsi updated the task description. (Show Details)
ayounsi added a subscriber: Cmjohnson.

Email sent to Equinix so they update their MAC filtering.

Router has been unracked and dropped at shipping for ship out. shipping information below

Papaul updated the task description. (Show Details)Oct 26 2018, 1:59 PM
Papaul closed this task as Resolved.Oct 26 2018, 2:06 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)
ayounsi reopened this task as Open.Oct 26 2018, 7:55 PM

Seems like we have a duplicate in Netbox:
https://netbox.wikimedia.org/dcim/devices/201/
and
https://netbox.wikimedia.org/dcim/devices/1954/

The 2nd one is more complete, so probably a matter of deleting the first one after verification.

Papaul closed this task as Resolved.Oct 30 2018, 2:38 AM

Double checked, it looks like https://netbox.wikimedia.org/dcim/devices/1954/ is complete so I deleted the first one