Looking at mw1267/1268 in rack A7: Can we move them from the bottom of the rack (above the ms-be) up to unit 20 and 27, staying in the same rack? That would be easiest for us.

https://netbox.wikimedia.org/dcim/racks/7/

Similarly for mw1313-1318, can they be moved into the free units higher up in the same rack? https://netbox.wikimedia.org/dcim/racks/15/

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:06:39Z] <mutante> depooled mw1267 (T266164)

Change 637572 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: replace proxy for eqiad A7, mw1268->mw1269

https://gerrit.wikimedia.org/r/637572

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:21:36Z] <mutante> replacing scap proxy for rack A7 eqiad because mw1268 needs to move physically (T266164)

Change 637572 merged by Dzahn:
[operations/puppet@production] scap: replace proxy for eqiad A7, mw1268->mw1269

https://gerrit.wikimedia.org/r/637572

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:35:20Z] <mutante> mw1268 - depooled for T266164

In T266164#6590028, @wiki_willy wrote:

++ @Cmjohnson for his feedback to @Dzahn 's question on the MW server moves

We talked on IRC and:

• mw1267,mw1268 need to move out of A7 completely
• we picked A8 as the destination
• too many servers to depool them all at once, we are starting with the 2 servers in A7, caring about the other rack later
• mw1267 is a regular appserver and depooled already
• mw1268 is special because it's a scap proxy, needed the extra change above and I just moved that role to mw1269, so that needs to stay where it is
• mw1268 also depooled now

Tomorrow Chris will move them to A8 and there will be another change needed to update comments in the repo and then we can pool them again.

mw13* will be later, tbd.

Change 637576 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: move mw1267,mw1268 from rack A7 to rack A8

https://gerrit.wikimedia.org/r/637576

Mentioned in SAL (#wikimedia-operations) [2020-10-30T14:14:16Z] <cmjohnson1> moving mw1267 and mw168 to rack A8 eqiad T266164

@Dzahn I moved mw1267 and 1268 to rack A8 and confirmed they're up. Updated netbox

Mentioned in SAL (#wikimedia-operations) [2020-10-30T20:59:17Z] <mutante> mw1267,mw1268 - scap pull and repool - back to prod - T266164

Change 637576 merged by Dzahn:
[operations/puppet@production] site: move mw1267,mw1268 from rack A7 to rack A8

https://gerrit.wikimedia.org/r/637576

One rack done, the other rack we will continue from week of Nov 16.

@Cmjohnson icinga reports that mw1267's mgmt is down, can you check? It also reports that PS redundancy is not good :(

mw1267 issues have been fixed

@Cmjohnson I'm back. we can continue with these second batch. Ideally in 2 batches. Let's talk what times are best.

@Cmjohnson So the servers at the bottom of B7 (mw1313-mw1318), should we move them to B4? That has the space and no other mw servers yet. Would that work for your purposes?

@Dzahn I would need to move them to a 1G rack, (B1,B3,B5,B6 and B8)

Mentioned in SAL (#wikimedia-operations) [2020-11-18T20:15:30Z] <mutante> mw1317,mw1318 - downtimed and depooled - they are physically moving from B7 to B5 (T266164)

Change 641834 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: update locations for mw1317, mw1318

https://gerrit.wikimedia.org/r/641834

Change 641834 merged by Dzahn:
[operations/puppet@production] site: update locations for mw1317, mw1318

https://gerrit.wikimedia.org/r/641834

@Dzahn

Is it possible to move
mw1281,82 and 83? I need this space for the an-workers on 10G. I can move them to A8.

I can definitely help on this @Dzahn, lemme know if you need a pair of extra hands :)

Icinga downtime for 4:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: move_to_other_rack

mw1281.eqiad.wmnet

Icinga downtime for 4:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: move_to_other_rack

mw1282.eqiad.wmnet

Icinga downtime for 4:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: move_to_other_rack

mw1283.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2020-12-09T18:05:49Z] <mutante> mw1281,mw1282,mw1283 shut down for T266164

@Cmjohnson Yes. I just depooled mw1281-1283, downtimed them and then shut them down physically. You can move them.

@Dzahn completed the move and mw1281-83 are up

@Cmjohnson Thank you. Repooled and receiving traffic again. Monitoring looks good.

Change 647326 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: update comment about location of mw1281-mw1283

https://gerrit.wikimedia.org/r/647326

Change 647326 merged by Dzahn:
[operations/puppet@production] site: update comment about location of mw1281-mw1283

https://gerrit.wikimedia.org/r/647326

@Cmjohnson Should this stay open for mw1313-mw1316 or did we solve the issue by moving other servers now?