refresh task: https://phabricator.wikimedia.org/T325215

As these servers are up for decom, they won't be migrated to k8s, and they are in the current secondary datacenter. It doesn't really matter to us in which rack you move them, so D8 is fine if it's the easiest for y'all. @Papaul How long do you estimate the move would take?

The move should be something like:
serviceops:

1. Depool the servers
2. Downtime the servers

ops-codfw:

3. Move the servers to their new location

serviceops:

4. Check the servers boot back ok and httpbb clears
5. Remove downtime
6. Repool the servers

Please tell us if something is missing from the above, and around what day/time you'd want to do that move, and we'll find someone available for it.

Change #1037466 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] apiserver: colocate API canary with scap proxy

https://gerrit.wikimedia.org/r/1037466

Change #1037466 merged by Hnowlan:

[operations/puppet@production] apiserver: colocate API canary with scap proxy

https://gerrit.wikimedia.org/r/1037466

mw2282 is a kubernetes server, so would need to be drained and cordoned as well. However since they are to be decommed and in the secondary datacenter, we don't need them for capacity, and are discussing decommissioning them so you don't have to move them. What would be the deadline for decision?

@Clement_Goubert thanks for the update. Since i can not edit your comment I updating it here.

The move should be something like:
serviceops:

1. Depool the servers
2. Downtime the servers
3. power down the servers

ops-codfw:

4. Move the servers to their new location
5. Re-cable the servers
6. configure you interface
7. Power back the servers

serviceops:

8. Check the servers boot back ok and httpbb clears
9. Remove downtime
10. Repool the servers

How long will this take? A maximum of 10 minutes per server but this can be less.
what would be the deadline for decision? If we can make it before June 15th that will be great. We are waiting on https://phabricator.wikimedia.org/T360671 and once that arrived things will start moving fast for the migration.

@Clement_Goubert hello just checking back with you to see if you have an update for me.

Thanks

So sorry I didn't answer earlier. Apart from mw2282 which has been migrated to k8s, we will decom these hosts so you don't have to move them. Just tell us when you want to move mw2282 and we'll drain/cordon/downtime it beforehand.

@Clement_Goubert next week monday 17th at 10:00am CT

All right, I'll do the draining monday beginning of the UTC afternoon so it's all set for you.

@Papaul All servers except mw2282 decommissioned.

@Clement_Goubert thank you.

rails, power, and network cables prepped for mw2282 move.

@Clement_Goubert hey we have a site visit at eqdfw on this Monday at 11:00am CT to check a power issue on our router that is there so we can not do the server move this Monday. Can you re-schedule that for this Wednesday 10:00am CT ? Thanks

Sure, no problem. Rescheduled.

Mentioned in SAL (#wikimedia-operations) [2024-06-19T13:17:14Z] <kamila_> drained mw2282.codfw.wmnet for T361856

Icinga downtime and Alertmanager silence (ID=8260b65f-a450-48bf-850e-4e458a18597f) set by kamila@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: host move

mw2282.codfw.wmnet

@Clement_Goubert it is a U.S holiday today can we please rescheduled this for tomorrow . Thank you Sorry about that

No worries, I'll extend the downtime, and we'll leave it like that for you to move.

Icinga downtime and Alertmanager silence (ID=111d8ee1-db67-4ba6-a57a-50da8c8dc4ff) set by cgoubert@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Host move

mw2282.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-06-20T16:00:38Z] <claime> Repooling and uncordoning mw2282.codfw.wmnet following move - T361856