Page MenuHomePhabricator

eqiad: Move logstash1020 to rack A8
Closed, ResolvedPublic

Description

@herron This is related to the an-worker task, I need to move this server and would like to do this 8 Feb @1530UTC. The server will stay in the same vlan just a different rack location. I intend to move to rack A8 ge-8/0/11

Please confirm this will work for you.

Event Timeline

elukey added a subscriber: elukey.

Added Filippo and Cole too for awareness. The idea is to shutdown the node, move it to a different rack within the same row (so no IP/vlan change) and boot it up again. The downtime requested will be around max 30 mins from my past experience, but it may also depend on how busy Chris is in the DC and if we have emergencies etc..

This will free rack space for Hadoop worker nodes so thanks a lot in advance for helping!

Hey @Cmjohnson, @elukey, sure this should be no problem. I've set a reminder in my calendar to stop services on this host ahead of the window, and yup as long as the host/network config stays the same ES should do the right thing when services are brought back up. Would like to monitor it as it comes up though, just shoot a ping when ready. Thanks!

Mentioned in SAL (#wikimedia-operations) [2021-02-08T14:50:30Z] <herron> stopped ES on logstash1020 in prep for re-rack T273984

@herron Thanks! all finished and I was able to ssh to the server.

logstash1020.mgmt is shown as down in icinga, reopening

logstash1020.mgmt 
View Service Details For This Host
DOWN	2021-02-09 10:08:30	0d 18h 33m 24s	1/2	PING CRITICAL - Packet loss = 100%
ayounsi reassigned this task from Cmjohnson to herron.
ayounsi added a subscriber: ayounsi.

The last Puppet run was at Mon Feb 8 14:16:19 UTC 2021 (19799 minutes ago). Puppet is disabled. disabled for re-racking T273984 --herron

Because of that it has been removed from PuppetDB and is alerting in the Netbox report (good safeguard!)

Please re-enable it if it's safe to do so and check that the Netbox report isn't alerting anymore https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/

Thanks @ayounsi it's been re-enabled and puppet has been run