Page MenuHomePhabricator

rack/setup/deploy new codfw mw app servers
Closed, ResolvedPublic

Description

This task will track the racking and setup/deployment of the 36 new mw app servers ordered on T134272
The racking schema will be defined after @Joe input in a different decommission task T135468. Re-used the same mgmt IP's , change the asset tag and use the same prod DNS IP's as well.

Rack A3 mw2215 to mw2238

  • receive in normally via T134272
  • racking
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (mw-raid1.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Joe)

Rack A4 mw2239 to mw2250

  • receive in normally via T134272
  • racking
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (mw-raid1.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Joe)

Event Timeline

Papaul created this task.May 17 2016, 5:06 AM
Restricted Application added a project: Operations. · View Herald TranscriptMay 17 2016, 5:06 AM
Restricted Application added subscribers: Zppix, Southparkfan, Aklapper. · View Herald Transcript
Papaul triaged this task as Medium priority.May 17 2016, 5:39 PM
RobH added a comment.EditedMay 17 2016, 6:50 PM

I'll toss my hat in the ring for wanting to test the service deployment step with our documentation.

It has been awhile since I pushed an apache from bare metal to service, so I'd like to re-familiarize myself with the process.

Papaul updated the task description. (Show Details)May 23 2016, 4:02 PM
Papaul updated the task description. (Show Details)May 24 2016, 5:41 PM
Papaul updated the task description. (Show Details)May 31 2016, 9:26 PM
Papaul updated the task description. (Show Details)Jun 1 2016, 3:36 AM
Papaul updated the task description. (Show Details)Jun 2 2016, 1:28 AM
Papaul added a comment.Jun 2 2016, 1:53 AM

@Joe I checked the DHCP file it looks like all the other mw app's servers are using Trusty. For the new mw app's server do you want me to install Trusty or switch to Jessie?

Thanks.

Papaul added a comment.Jun 6 2016, 7:27 PM

I chat with Joe on IRC he said to install Jessie on all the new mw app servers

Joe added a comment.Jun 7 2016, 10:57 AM

A few notes:

  • All machines have double disks, so we will use the mw-raid1.cfg partman recipe here, I will probably prepare a patch for that.
  • The servers need to be added to site.pp as well, in the correct groups; I'll work on that
  • After we've done that, we will need to follow a relatively simple procedure - it's mostly delineated on a pad https://etherpad.wikimedia.org/p/jessie-install
Papaul updated the task description. (Show Details)Jun 7 2016, 11:46 PM
Papaul updated the task description. (Show Details)Jun 8 2016, 12:46 AM
Papaul updated the task description. (Show Details)Jun 8 2016, 12:51 AM
Papaul updated the task description. (Show Details)Jun 9 2016, 12:19 AM
Dzahn added a subscriber: Dzahn.Jun 9 2016, 6:13 AM

These have been added to DNS now , mgmt and prod IPs, changes by papaul, i reviewed and merged. papaul started installing servers ..mw2215 thru mw2219 already showing up in Icinga and i acked some alerts from icinga about them not being in dsh groups yet (normal)

Dzahn added a comment.Jun 10 2016, 5:05 AM

DHCP: Add MAC address for mw2218 and mw2239 to mw2250 https://phabricator.wikimedia.org/T135466 merged, mw2243 was not in it but i assumed papaul had a reason to skip that

Papaul updated the task description. (Show Details)Jun 10 2016, 3:27 PM

OS install, puppet certs, salt-key, initial run done on mw2215-mw2232

Change 294023 had a related patch set uploaded (by Giuseppe Lavagetto):
mediawiki: correctly assign the new codfw appservers in puppet

https://gerrit.wikimedia.org/r/294023

Change 294023 merged by Giuseppe Lavagetto:
mediawiki: correctly assign the new codfw appservers in puppet

https://gerrit.wikimedia.org/r/294023

Mentioned in SAL [2016-06-14T11:04:00Z] <_joe_> pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466)

Papaul reassigned this task from Papaul to Joe.Jun 21 2016, 2:51 PM
Papaul updated the task description. (Show Details)

OS installation complete on all the hosts puppet cert and salt-key complete as well.

Joe closed this task as Resolved.Jul 28 2016, 10:05 AM