Page MenuHomePhabricator

eqiad: Physical moves for MediaWiki servers
Open, MediumPublicRequest

Description

Hi @wkandek - in preparation for an upcoming install for some very heavy ms-be servers, @Cmjohnson is looking to free up some rack space at the bottom of our cabinets to install them. Would it be possible if you could designate someone from Service-Ops to work with Chris on having the following servers moved to a different rack location at eqiad?

  • mw1313
  • mw1314
  • mw1315
  • mw1316
  • mw1317
  • mw1318
  • mw1267
  • mw1268

The initial timeframe that we're thinking is probably sometime during the first week of November., but that can be adjusted based on availability. Much appreciated in advance.

Thanks,
Willy

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenRequestDzahn

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptOct 21 2020, 5:31 PM
wiki_willy added a parent task: Unknown Object (Task).Oct 21 2020, 5:33 PM
Dzahn added a subscriber: Dzahn.Thu, Oct 29, 9:35 PM

Looking at mw1267/1268 in rack A7: Can we move them from the bottom of the rack (above the ms-be) up to unit 20 and 27, staying in the same rack? That would be easiest for us.

https://netbox.wikimedia.org/dcim/racks/7/

Dzahn added a comment.Thu, Oct 29, 9:42 PM

Similarly for mw1313-1318, can they be moved into the free units higher up in the same rack? https://netbox.wikimedia.org/dcim/racks/15/

wiki_willy reassigned this task from wkandek to Cmjohnson.Thu, Oct 29, 9:44 PM

++ @Cmjohnson for his feedback to @Dzahn 's question on the MW server moves

Looking at mw1267/1268 in rack A7: Can we move them from the bottom of the rack (above the ms-be) up to unit 20 and 27, staying in the same rack? That would be easiest for us.
https://netbox.wikimedia.org/dcim/racks/7/

Similarly for mw1313-1318, can they be moved into the free units higher up in the same rack?
https://netbox.wikimedia.org/dcim/racks/15/

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:06:39Z] <mutante> depooled mw1267 (T266164)

Change 637572 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: replace proxy for eqiad A7, mw1268->mw1269

https://gerrit.wikimedia.org/r/637572

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:21:36Z] <mutante> replacing scap proxy for rack A7 eqiad because mw1268 needs to move physically (T266164)

Change 637572 merged by Dzahn:
[operations/puppet@production] scap: replace proxy for eqiad A7, mw1268->mw1269

https://gerrit.wikimedia.org/r/637572

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:35:20Z] <mutante> mw1268 - depooled for T266164

++ @Cmjohnson for his feedback to @Dzahn 's question on the MW server moves

We talked on IRC and:

  • mw1267,mw1268 need to move out of A7 completely
  • we picked A8 as the destination
  • too many servers to depool them all at once, we are starting with the 2 servers in A7, caring about the other rack later
  • mw1267 is a regular appserver and depooled already
  • mw1268 is special because it's a scap proxy, needed the extra change above and I just moved that role to mw1269, so that needs to stay where it is
  • mw1268 also depooled now

Tomorrow Chris will move them to A8 and there will be another change needed to update comments in the repo and then we can pool them again.

mw13* will be later, tbd.

Change 637576 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: move mw1267,mw1268 from rack A7 to rack A8

https://gerrit.wikimedia.org/r/637576

Mentioned in SAL (#wikimedia-operations) [2020-10-30T14:14:16Z] <cmjohnson1> moving mw1267 and mw168 to rack A8 eqiad T266164

@Dzahn I moved mw1267 and 1268 to rack A8 and confirmed they're up. Updated netbox

Dzahn claimed this task.Fri, Oct 30, 8:58 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-30T20:59:17Z] <mutante> mw1267,mw1268 - scap pull and repool - back to prod - T266164

Change 637576 merged by Dzahn:
[operations/puppet@production] site: move mw1267,mw1268 from rack A7 to rack A8

https://gerrit.wikimedia.org/r/637576

Dzahn updated the task description. (Show Details)Tue, Nov 3, 1:57 AM

One rack done, the other rack we will continue from week of Nov 16.

Dzahn moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Tue, Nov 3, 2:15 AM
Dzahn moved this task from Doing 😎 to Next up 🥌 on the serviceops board.
Krinkle renamed this task from eqiad: Physical Moves for MediaWiki Servers to eqiad: Physical moves for MediaWiki servers.Tue, Nov 3, 4:17 AM
elukey added a subscriber: elukey.Wed, Nov 4, 7:01 AM

@Cmjohnson icinga reports that mw1267's mgmt is down, can you check? It also reports that PS redundancy is not good :(

mw1267 issues have been fixed

Dzahn added a comment.Tue, Nov 17, 1:51 AM

@Cmjohnson I'm back. we can continue with these second batch. Ideally in 2 batches. Let's talk what times are best.

Dzahn added a comment.EditedTue, Nov 17, 1:59 AM

@Cmjohnson So the servers at the bottom of B7 (mw1313-mw1318), should we move them to B4? That has the space and no other mw servers yet. Would that work for your purposes?

@Dzahn I would need to move them to a 1G rack, (B1,B3,B5,B6 and B8)

Mentioned in SAL (#wikimedia-operations) [2020-11-18T20:15:30Z] <mutante> mw1317,mw1318 - downtimed and depooled - they are physically moving from B7 to B5 (T266164)

Dzahn updated the task description. (Show Details)Wed, Nov 18, 8:53 PM
Dzahn updated the task description. (Show Details)

Change 641834 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: update locations for mw1317, mw1318

https://gerrit.wikimedia.org/r/641834

Change 641834 merged by Dzahn:
[operations/puppet@production] site: update locations for mw1317, mw1318

https://gerrit.wikimedia.org/r/641834