Page MenuHomePhabricator
Create Task
Maniphest T135466

rack/setup/deploy new codfw mw app servers
Closed, ResolvedPublic
Actions

Description

This task will track the racking and setup/deployment of the 36 new mw app servers ordered on T134272
The racking schema will be defined after @Joe input in a different decommission task T135468. Re-used the same mgmt IP's , change the asset tag and use the same prod DNS IP's as well.

Rack A3 mw2215 to mw2238

  • receive in normally via T134272
  • racking
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (mw-raid1.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Joe)

Rack A4 mw2239 to mw2250

  • receive in normally via T134272
  • racking
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (mw-raid1.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Joe)

Details

SubjectRepoBranchLines +/-
mediawiki: correctly assign the new codfw appservers in puppetoperations/puppetproduction+29 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

Papaul created this task.May 17 2016, 5:06 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 17 2016, 5:06 AM
Restricted Application added subscribers: Zppix, Southparkfan, Aklapper. · View Herald Transcript
Papaul created subtask T135468: codfw old mw app server decomission .May 17 2016, 5:28 AM
Papaul updated the task description. (Show Details)
Papaul triaged this task as Medium priority.May 17 2016, 5:39 PM
RobH added a comment.EditedMay 17 2016, 6:50 PM

I'll toss my hat in the ring for wanting to test the service deployment step with our documentation.

It has been awhile since I pushed an apache from bare metal to service, so I'd like to re-familiarize myself with the process.

Papaul updated the task description. (Show Details)May 23 2016, 4:02 PM
Papaul updated the task description. (Show Details)May 24 2016, 5:41 PM
Papaul updated the task description. (Show Details)May 31 2016, 9:26 PM
Papaul updated the task description. (Show Details)Jun 1 2016, 3:36 AM
Papaul created subtask T136670: rack/setup/deploy mw22[1-5][0-9] switch configuration .Jun 1 2016, 3:51 AM
Papaul updated the task description. (Show Details)Jun 2 2016, 1:28 AM
Papaul added a comment.Jun 2 2016, 1:53 AM

@Joe I checked the DHCP file it looks like all the other mw app's servers are using Trusty. For the new mw app's server do you want me to install Trusty or switch to Jessie?

Thanks.

Papaul added a comment.Jun 6 2016, 7:27 PM

I chat with Joe on IRC he said to install Jessie on all the new mw app servers

Joe added a comment.Jun 7 2016, 10:57 AM

A few notes:

  • All machines have double disks, so we will use the mw-raid1.cfg partman recipe here, I will probably prepare a patch for that.
  • The servers need to be added to site.pp as well, in the correct groups; I'll work on that
  • After we've done that, we will need to follow a relatively simple procedure - it's mostly delineated on a pad https://etherpad.wikimedia.org/p/jessie-install
Dzahn mentioned this in rOPUP1c8f6843421e: DHCP: Add MAC address entries for mw2215-mw2238 skipping mw2218 Bug:T135466.Jun 7 2016, 10:15 PM
Papaul updated the task description. (Show Details)Jun 7 2016, 11:46 PM
Papaul updated the task description. (Show Details)Jun 8 2016, 12:46 AM
Papaul updated the task description. (Show Details)Jun 8 2016, 12:51 AM
Papaul updated the task description. (Show Details)Jun 9 2016, 12:19 AM
Dzahn subscribed.Jun 9 2016, 6:13 AM

These have been added to DNS now , mgmt and prod IPs, changes by papaul, i reviewed and merged. papaul started installing servers ..mw2215 thru mw2219 already showing up in Icinga and i acked some alerts from icinga about them not being in dsh groups yet (normal)

Dzahn added a comment.Jun 9 2016, 6:14 AM

https://gerrit.wikimedia.org/r/#/c/293446/
https://gerrit.wikimedia.org/r/#/c/292307/
https://gerrit.wikimedia.org/r/#/c/293246/
https://gerrit.wikimedia.org/r/#/c/293218/
https://gerrit.wikimedia.org/r/#/c/292072/

Dzahn mentioned this in rOPUP00354a25f70e: DHCP: Add MAC address for mw2218 and mw2239 to mw2250 Bug:T135466.Jun 10 2016, 5:04 AM
Dzahn added a comment.Jun 10 2016, 5:05 AM

DHCP: Add MAC address for mw2218 and mw2239 to mw2250 https://phabricator.wikimedia.org/T135466 merged, mw2243 was not in it but i assumed papaul had a reason to skip that

Papaul updated the task description. (Show Details)Jun 10 2016, 3:27 PM
Papaul added a comment.Jun 10 2016, 3:40 PM

OS install, puppet certs, salt-key, initial run done on mw2215-mw2232

gerritbot subscribed.Jun 13 2016, 5:44 AM

Change 294023 had a related patch set uploaded (by Giuseppe Lavagetto):
mediawiki: correctly assign the new codfw appservers in puppet

https://gerrit.wikimedia.org/r/294023

gerritbot added a project: Patch-For-Review.Jun 13 2016, 5:44 AM
gerritbot added a comment.Jun 13 2016, 10:21 AM

Change 294023 merged by Giuseppe Lavagetto:
mediawiki: correctly assign the new codfw appservers in puppet

https://gerrit.wikimedia.org/r/294023

Joe mentioned this in rOPUPad5bde0929bf: mediawiki: correctly assign the new codfw appservers in puppet.Jun 13 2016, 10:24 AM
Stashbot subscribed.Jun 14 2016, 11:04 AM

Mentioned in SAL [2016-06-14T11:04:00Z] <_joe_> pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466)

Papaul mentioned this in rOPUPe757e3c7a91e: DHCP: Add mw2243 MAC address Bug:T135466.Jun 17 2016, 6:05 PM
Joe mentioned this in rOPUP20eaa46e9c6d: mediawiki: correctly assign the new codfw appservers in puppet.Jun 17 2016, 6:05 PM
Joe mentioned this in rOPUP1d9afd89347b: mediawiki: correctly assign the new codfw appservers in puppet.
Papaul mentioned this in rOPUP4514bb8bfa93: DHCP: Add MAC address for mw2218 and mw2239 to mw2250 Bug:T135466.Jun 17 2016, 6:05 PM
Dzahn mentioned this in rOPUPcf50a7badbc4: DHCP: Add MAC address entries for mw2215-mw2238 skipping mw2218 Bug:T135466.Jun 17 2016, 6:06 PM
Papaul mentioned this in rOPUP0dc6467a618c: DHCP: Add MAC address entries for mw2215-mw2238 skipping mw2218 Bug:T135466.Jun 17 2016, 6:06 PM
elukey mentioned this in rOPUP691ed340fa59: DHCP: Add mw2243 MAC address Bug:T135466.Jun 20 2016, 3:55 PM
Papaul reassigned this task from Papaul to Joe.Jun 21 2016, 2:51 PM
Papaul updated the task description. (Show Details)

OS installation complete on all the hosts puppet cert and salt-key complete as well.

Joe closed this task as Resolved.Jul 28 2016, 10:05 AM
RobH closed subtask T135468: codfw old mw app server decomission as Resolved.Jan 19 2017, 2:15 AM
RobH closed subtask T136670: rack/setup/deploy mw22[1-5][0-9] switch configuration as Resolved.May 31 2017, 8:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL