Page MenuHomePhabricator

(Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet
Open, HighPublic


This task will track the racking, setup, and installation of 36 new mw systems ordered via T228214.

Racking Proposal: Due to the recent order of additional mw hosts on upcoming task T233639, the racking for this needs to be re-evaluated by @Joe and team to ensure they get the best placement for this batch of 36 and the other batch of 23 systems incoming. This section will be updated by them.

Individual server setup checklist (this checklist should be duplicated for each system):

  • - receive in system on procurement task T228214.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'


Related Gerrit Patches:

Event Timeline

RobH triaged this task as Medium priority.Oct 24 2019, 8:39 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptOct 24 2019, 8:39 PM
RobH assigned this task to Joe.Oct 24 2019, 8:40 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

IRC update: Chatted with @Joe and I'm assigning this to him first so he can evaluate where he wants to have all of these incoming mw hosts racked. He'll update this task, and please then reassign to @Jclark-ctr for him to start racking the servers.


wiki_willy reassigned this task from Joe to Jclark-ctr.Oct 31 2019, 7:41 PM
wiki_willy added a subscriber: wiki_willy.

Assigning to @Jclark-ctr since he's going to be taking care of the install, but @Joe - let us know if there are any specific racking instructions for these. Thanks, Willy

Joe added a comment.Nov 5 2019, 6:21 AM

Sorry I'm lagging behind on this. Point is, I shouldn't be the only person to answer this question but most of my team is out this week. I'll try to reassign this to someone with time to respond.

@Joe et al - just following up on this, since we've had these servers onsite for a while now. Also, once you have any specific racking instructions for us, please keep in mind that we'll need some time to complete all the tasks associated with getting these hosts set up, so it should be reflected in your requested "need by date" as well.


wiki_willy renamed this task from rack/setup/install mw13[49-84].eqiad.wmnet to (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet.Nov 22 2019, 8:57 PM
jijiki added a subscriber: jijiki.Nov 22 2019, 10:07 PM

@wiki_willy I will provide racking instructions on Monday for you, sorry we have delayed you this much.

Thanks @jijiki , much appreciated


RackA 5A6A 7B 6B 7C 6D 4D 5
mw servers6617216306 (decom)30 (decom)

We will decommission 36 servers from D 4 and D 5.

I recommend we add:

  • D 1: 10 servers
  • D 2: 6 servers
  • D 6: 10 servers
  • D 8: 10 servers

Regarding the 23 upcoming ones, we will split them between rows A, B, C. Please let me know if that rack setup makes sense. Thank you!

jijiki added a comment.Dec 5 2019, 4:46 PM

@Jclark-ctr Do we have an estimate when we are able to have those servers racked?

Hi @jijiki - we received these on Sept 25, but just recently received the racking instructions from your team on Nov 25, so there might be some shuffling around we need to do, before getting these racked to meet the recommendations. When do you need these handed over to you by? And we can try to meet that need by date. Thanks, Willy

jijiki added a comment.Dec 5 2019, 5:29 PM

@wiki_willy given we are responsible for this delay, we would like to check with your team and tell us what is the earliest we can do, and we can in turn can plan around that. Sorry for the delays on our end. I would like to avoid requesting a date that is unrealistic, given that xmas is round the corner. Thank you!

Thanks for confirming @jijiki - I'll let @Jclark-ctr provide an ETA on them, when he gets in a bit later today. Thanks, Willy

@jijiki If no surprises i could have them Racked by Dec 20th

jijiki added a comment.Dec 5 2019, 5:43 PM

That would be lovely, thank you!

wiki_willy renamed this task from (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet to (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.Dec 5 2019, 5:46 PM
Papaul added a subscriber: Papaul.Dec 6 2019, 1:18 AM

@Jclark-ctr Please see below for available mgmt IP's that you can use for those servers. Once you have the asset tags please update the table with the information and we can enter the the name, ip and asset tag in the DNS file. Let me know if you have any questions. Thanks

mgmt IP's

HostIPAsset tag


I recommend we use these racks we are limited in space . In row D and would not have the enough sfp to rj45 to accommodate the putting these in 10g racks
D 1: 15 servers
D 3: 3 servers
D 6: 15 servers
D 8: 3 servers

@Jclark-ctr if you feel this will work better, we are happy. Either way, this racking is still better than the original one (30 servers in D 5). Thank you!

@jijiki small delay had a few tickets become urgent. Being a contractor i will be in week after christmas and will wrap them up at beginning of new year.

Hey @Jclark-ctr ! Do we have an update about when we will have those servers ready? Thank you!

jijiki added a subscriber: Cmjohnson.EditedWed, Jan 22, 4:21 PM

@Jclark-ctr @Cmjohnson @wiki_willy please let us know the status of racking these servers and update the task accordingly, thank you!

Jclark-ctr updated the task description. (Show Details)

Host racked bios, ip , and password set. Needs dns

server ip asset tag rack switch port
mw1349 WMF5291 D1 42
mw1350 WMF5292 D1 41
mw1351 WMF5293 D1 40
mw1352 WMF5294 D1 39
mw1353 WMF5295 D1 38
mw1354 WMF5296 D1 37
mw1355 WMF5297 D1 36
mw1356 WMF5298 D1 35
mw1357 WMF5299 D1 33
mw1358 WMF5300 D1 34
mw1359 WMF5301 D1 29
mw1360 WMF5302 D1 30
mw1361 WMF5303 D1 31
mw1362 WMF5304 D1 32
mw1363 WMF5305 D3 40
mw1364 WMF5306 D3 41
mw1365 WMF5307 D3 42
mw1366 WMF5308 D6 27
mw1367 WMF5309 D6 26
mw1368 WMF5310 D6 25
mw1369 WMF5311 D6 24
mw1370 WMF5312 D6 23
mw1371 WMF5313 D6 22
mw1372 WMF5314 D6 21
mw1373 WMF5315 D6 20
mw1374 WMF5316 D6 19
mw1375 WMF5317 D6 18
mw1376 WMF5318 D6 17
mw1377 WMF5319 D6 16
mw1378 WMF5320 D6 15
mw1379 WMF5321 D6 14
mw1380 WMF5322 D6 13
mw1381 WMF5323 D6 12
mw1382 WMF5324 D6 11
mw1383 WMF5325 D8 25
mw1384 WMF5326 D8 26

Joe raised the priority of this task from Medium to High.Tue, Feb 4, 7:53 AM

Can we please expedite this?

We really need these servers to join rotation.

@Joe working on these next...will have to you in next day or 2.

mark added a subscriber: mark.Wed, Feb 12, 4:47 PM

@wiki_willy With Chris having been ill the past few days, what's a realistic new ETA for this?

@mark - just chatted with Chris and he's working on them now (he's back today after being sick), so ETA is end of day. Thanks, Willy

RobH removed a subscriber: RobH.Wed, Feb 12, 5:32 PM

Change 571782 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

Change 571782 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

Change 571785 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

Change 571785 merged by Cmjohnson:
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

Change 571797 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

Cmjohnson updated the task description. (Show Details)Wed, Feb 12, 8:45 PM

@Jclark-ctr when you're at data center later today could you look into these please.

mw1351 and mw1381 have the wrong password set
mw1379 and mw1380 are not getting a connection, most likely cable related.

Change 571797 merged by Cmjohnson:
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

Latest update: there were some settings that need to be redone, so @Cmjohnson will get these fixed tomorrow, with Thursday as the new ETA. Apologies for the long delay. Thanks, Willy

Change 572031 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac to dhcpd for mw1351

Change 572031 merged by Cmjohnson:
[operations/puppet@production] Adding mac to dhcpd for mw1351

Change 572066 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

Change 572066 merged by Cmjohnson:
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

Cmjohnson reassigned this task from Cmjohnson to Joe.Thu, Feb 13, 7:47 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson removed a project: ops-eqiad.

@Joe These are ready for you now, initial puppet run has been completed. I am removing from ops-eqiad tag. If you need any additional on-site work please add tag back and ping me. assigning to you