Page MenuHomePhabricator

replace ulsfo access switches
Closed, ResolvedPublic

Description

This task will track the onsite and software configuration of two new access switches for ulsfo, purchased on T169308.

Both of these switches are currently sitting in the racks (not racked) @ ulsfo.

The target date for this work is Wednesday, January 31st, 2018.

  • - both new switches in racktables new-asw1-ulsfo & new-asw2-ulsfo
  • - @ayounsi confirms he has a valid backup and all config info needed for migration
  • - write down the unique serial # labeled on each fiber optic cable in each network port. This will give us the fastest 1:1 port mapping for re-connection, since we don't list the cable #s in the switch description on servers in ulsfo (though it wouldn't be a bad idea to append them after we've documented them for this migration.)
  • - ulsfo depooled from serving traffic via dns
  • - set ALL ulsfo servers to maint mode (for server/mgmt/and all service checks) in icinga
  • - depool all cp systems via confctl
  • - unplug all fibers/optics from old asw switches (one at a time), unrack old switch, and rack new switch in its place (use the rackstuds that are in the in rack storage bin)
  • - configure new switch stack, verify networking configuration
  • - babysit icinga/ssh checks to ensure connectivity to all systems
  • - check with traffic, recovery of cp systems may require extra steps based on how long systems were unreachable.
  • - traffic team review and repooling of ulsfo via dns

Details

Related Gerrit Patches:

Event Timeline

RobH triaged this task as High priority.Jan 18 2018, 5:11 PM
RobH created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2018, 5:11 PM
RobH added a comment.EditedJan 18 2018, 5:11 PM

Since this involves Traffic as well as netops, this plan should get @BBlack's review/approval.

The target date for this work is Wednesday, January 31st, 2018.

RobH updated the task description. (Show Details)Jan 18 2018, 5:14 PM
ayounsi updated the task description. (Show Details)Jan 18 2018, 5:27 PM
ema moved this task from Triage to Hardware on the Traffic board.Jan 19 2018, 5:43 PM

Change 407022 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] depooling ulsfo to swap switches

https://gerrit.wikimedia.org/r/407022

Change 407022 merged by RobH:
[operations/dns@master] depooling ulsfo to swap switches

https://gerrit.wikimedia.org/r/407022

RobH updated the task description. (Show Details)Jan 31 2018, 6:47 PM

Change 407059 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Renaming asw-ulsfo to asw2-ulsfo in Icinga, Smokeping and Rancid

https://gerrit.wikimedia.org/r/407059

Change 407059 merged by Ayounsi:
[operations/puppet@production] Renaming asw-ulsfo to asw2-ulsfo in Icinga, Smokeping and Rancid

https://gerrit.wikimedia.org/r/407059

RobH updated the task description. (Show Details)
RobH assigned this task to BBlack.Jan 31 2018, 9:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T23:06:07Z] <bblack> re-pooling ulsfo in DNS - T185228

RobH closed this task as Resolved.Feb 8 2018, 10:20 PM