@Papaul the databases and es2029/es2030 are ready for relocation. Please turn them ON once you are done
For what is worth, es2029 and es2030 are scheduled to be done 14th, which is a bank holiday for me, so someone else would need to bring mysql up once that is done.

In T305469#7843940, @Marostegui wrote:

For what is worth, es2029 and es2030 are scheduled to be done 14th, which is a bank holiday for me, so someone else would need to bring mysql up once that is done.

I can take care of that.

Thanks @Kormat

mysql started on db* hosts

mysql started on es* hosts

Changing the tag as our DBA part here is done. If there's anything else required, I am still subscribed to the task.

@Marostegui thanks

Icinga downtime and Alertmanager silence (ID=dc2c981d-aef2-4a2b-9d24-2e3ca912b985) set by bking@cumin1001 for 1 day, 0:00:00 on 1 host(s) and their services with reason: physically moving host

wcqs2001.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=9620461c-f770-40dd-99d6-2b4f895a2549) set by akosiaris@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: moving to a different rack

rdb2008.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=6e1e84ea-fac8-4dde-be55-1bf6ea935f75) set by akosiaris@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: moving to a different rack

kubestage2002.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=5293ea70-e1a3-4862-ae77-82e8abf9cdd4) set by akosiaris@cumin1001 for 2:00:00 on 1 host(s) and their services with reason: moving to a different rack

mc2023.codfw.wmnet

I marked rdb2008, kubestage2002 and mc2023 as YES in the table. rdb2008 is the secondary, not the primary, kubestage2002 is for the staging cluster anyway and mc2023 will be handled by mcrouter's configuration and shard05 should be moved to mc-gp* hosts (gutter pool).

I can not power down kuberstage2002

W: aborting poweroff due to 30-query-hostname exiting with code 1.

@hnowlan will it be possible to get me restbase2021 offline on April 14th at 9:30am CT?

thanks.

In T305469#7845959, @Papaul wrote:

@hnowlan will it be possible to get me restbase2021 offline on April 14th at 9:30am CT?

thanks.

Yep, that should be fine - it ideally shouldn't be down for too long but please ping me when ready and I can take it down.

Icinga downtime and Alertmanager silence (ID=575a5fd0-668b-41f6-8ab3-5ff749f54ac7) set by akosiaris@cumin1001 for 2 days, 0:00:00 on 1 host(s) and their services with reason: moving to a different rack

mc2023.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=60f8ccbd-38ba-4b65-aadf-f44a7fc83c9e) set by akosiaris@cumin1001 for 2 days, 0:00:00 on 1 host(s) and their services with reason: moving to a different rack

kubestage2002.codfw.wmnet

@Papaul: mc2023 and kubestage2002 have been downtimed again (for 2days) and I 've just powered them off. The should be ready to be moved.

@akosiaris thanks will move them tomorrow.

@Andrew @aborrero I have listed 14 servers that we will have to move into rack b1 4 of those are not in row B and using Public IP. I think will be better to do those in after moving the others that are already in row B.
I am thinking on starting the move next Monday April 18th. what are the servers that are in row B and not in Rack B1 that we can move next Monday and how many can we do that same day.(for me if we can do all 10 that Monday, I have no issues) if Monday is not a good day please let me know. Please add in the "notes" section a comment if we can move that server next Monday. Thanks

@hnowlan i am ready for restbase2021

In T305469#7855361, @Papaul wrote:

@hnowlan i am ready for restbase2021

Go ahead!

All the nodes that are not cloud are now out of Rack B1. Thanks to all helping me to de-pool the servers and power them off.

I have a dentist appointment at 2PM CDT on Monday the 18th; otherwise I'm available to help with this.

Please be aware that I'm largely ignorant of network topology vs. racks so will be relying on papaul or netops staff to ensure that this move keeps everything still able to talk to everything else.

Mentioned in SAL (#wikimedia-cloud) [2022-04-18T13:40:05Z] <andrewbogott> shutting down many codfdfw1dev servers (including network infra!) for T305469

backups are complaining of lack of recent backups of cloudcontrol2003-dev, as it is down. I will ignore those for a while- we must reenable monitoring once maintenance completes.

Change 784621 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Ignore cloudcontrol2003-dev backup monitoring

https://gerrit.wikimedia.org/r/784621

Change 784621 merged by Jcrespo:

[operations/puppet@production] backup: Ignore cloudcontrol2003-dev backup monitoring

https://gerrit.wikimedia.org/r/784621