Page MenuHomePhabricator

refresh/replace scs-ulsfo
Closed, ResolvedPublic0 Story Points

Description

This task will track the replacement of the scs-ulsfo with the future-scs-ulsfo (updated model with dual power input and does away with the requirement of cross over cables.)

Since this does away with cross over cables, we will need to order some replacement orange patch cables to replace the ones listed below:

port #devicelength of replacement orange patch cable (in feet)
1cr3-ulsfo3
2asw1-ulsfo3
3mr1-ulsfo2
4cr4-ulsfo6
5asw2-ulsfo6
6ps-103.02.224
7ps-103.02.236
8atlas-ulsfo (not connected yet)6

PLANNED MAINTENANCE WINDOW IS 2019-08-14 @ ~18:00 GMT to ~20:00 GMT

Work checklist:

  • - plan unavailability to serial/mgmt to all the devices listed above - email team list with details
  • - save scs-ulsfo config to local laptop for migration/restoration
  • - wipe config of existing scs-ulsfo and remove it from the rack (update netbox status and racking assignment)
  • - install new scs-ulsfo in old rack U
  • - restore old config, test that scs console is online, login works, old ports are listed
  • - update to syslog.anycast.wmnet and dns to 10.3.0.1
  • - connect all new orange patch cables, update netbox with the label numbers/assignments
connection checklist

If checked off, a system has a new cable run, labeled, updated in netbox, and tested as working.

  • - cr3-ulsfo
  • - asw1-ulsfo
  • - mr1-ulsfo
  • - cr4-ulsfo
  • - asw2-ulsfo
  • - ps-103.02.22
  • - ps-103.02.23
  • - atlas-ulsfo

Event Timeline

RobH triaged this task as Normal priority.Aug 7 2019, 10:47 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptAug 7 2019, 10:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH updated the task description. (Show Details)Aug 7 2019, 10:55 PM
RobH mentioned this in Unknown Object (Task).Aug 7 2019, 10:58 PM
RobH added a subtask: Unknown Object (Task).
RobH added a comment.Aug 13 2019, 4:20 PM

The cables have arrived for this. I'll go onsite on Wednesday, August 14th to swap out the scs-ulsfo console server.

I'll email the SRE team list to ensure the department is aware of the change/downtime. This won't affect anything, unless someone is depending on serial connections during the work (hence the email.)

RobH updated the task description. (Show Details)Aug 13 2019, 4:22 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-14T22:01:07Z] <robh> starting scs-ulsfo replacement. There will be icinga errors and they are intentionally being allowed so we know when things dont recover properly T230077

Mentioned in SAL (#wikimedia-operations) [2019-08-15T00:15:08Z] <robh> scs-ulsfo offline due to networking issues, rob returning tomorrow with fix T230077

RobH updated the task description. (Show Details)Aug 15 2019, 7:11 PM
RobH updated the task description. (Show Details)Aug 15 2019, 7:32 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Aug 15 2019, 7:37 PM
RobH updated the task description. (Show Details)Aug 15 2019, 7:41 PM
RobH closed this task as Resolved.Aug 15 2019, 8:07 PM
RobH updated the task description. (Show Details)

Ok, the new scs is now in place, with all connections documented and tested as working.

RobH closed subtask Unknown Object (Task) as Resolved.Aug 16 2019, 9:03 PM