Page MenuHomePhabricator

relocate/reimage cloudvirt1013 with 10G interfaces
Closed, ResolvedPublic

Description

cloudvirt1013:

  • - put system offline in all checks for maint window
  • - relocate to 10G rack and update netbox
  • - update switch configuration for new primary 10G
  • - enable PXE for 10G primary interface.

[]x - attach/cable secondary 10G port for instance traffic, update switch config.

  • - remove old switch config for 1G ports
  • - (update firmware?)
  • - PXE boot and reimage system
  • - reintroduce system into service cluster

Event Timeline

Andrew changed the task status from Open to Stalled.Feb 4 2020, 5:41 PM

Ah, dammit, dc-ops missed this ticket and now 1013 is back in service on 1G. So it's no longer a good time to do this, there's real workload on that host.

@Andrew This is already located in 10g rack just needs Dac cables and connected to 10g nic. I have talked to @JHedden regarding this not sure if additional changes with switch is needed if using same ports.

@Andrew not sure if you noticed my last comment?

@jclark, typically we need to drain the workload from a host before we can swap it. It was empty when I opened this task but no longer, so this is blocked until we have a good way of draining it (and a good place to move the workload.)

Mentioned in SAL (#wikimedia-cloud) [2020-10-06T21:30:09Z] <andrewbogott> moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for T243414

Andrew raised the priority of this task from Low to Medium.
Andrew added a subscriber: Jclark-ctr.

@Cmjohnson , This host is empty again; you can power it down any time. I'm happy to do the re-imaging steps but do please check the firmware.

Thank you!

From the parent task:

Note that now racks C8 and D5 are dedicated to WMCS servers (including cloudvirt). So please move servers there when able.

Andrew changed the task status from Stalled to Open.Oct 19 2020, 4:30 PM

Change 635324 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding new mac address for cloudvirt1013

https://gerrit.wikimedia.org/r/635324

Change 635324 merged by Cmjohnson:
[operations/puppet@production] Adding new mac address for cloudvirt1013

https://gerrit.wikimedia.org/r/635324

@Andrew The server didn't have to move locations, added 10G cables, fixed the network switch, updated dhcpd file with new mac address. Verified the server will pxe boot off the 10G card now. I did not update f/w

Thanks! I'll re-image and see what I can see.

Change 635370 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1013: switch nova to use the 10G nics

https://gerrit.wikimedia.org/r/635370

Change 635370 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1013: switch nova to use the 10G nics

https://gerrit.wikimedia.org/r/635370

Andrew updated the task description. (Show Details)