Page MenuHomePhabricator
Create Task
Maniphest T132976

rack/setup/deploy restbase200[7-9]
Closed, ResolvedPublic
Actions

Description

The systems for this task have been ordered on procurement task T130218. Once these arrive on-site, they will be racked in A5, C5 and D5

restbase2007

  • receive in normally via T130218
  • rack in b1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

restbase2008

  • receive in normally via T130218
  • rack in c1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

restbase2009

  • receive in normally via T130218
  • rack in d1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

Details

SubjectRepoBranchLines +/-
restbase: add restbase200[789] to conftool-dataoperations/puppetproduction+3 -0
cassandra: add restbase2008-boperations/puppetproduction+5 -5
cassandra: add restbase2009-boperations/puppetproduction+5 -5
cassandra: add restbase200[789]operations/puppetproduction+64 -1
cassandra: add restbase200[789] instancesoperations/dnsmaster+19 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Papaul created this task.Apr 18 2016, 10:18 PM

@fgiunchedi pleae confirm that the setup and racking information for the new restbase servers are correct. Thanks

Papaul triaged this task as High priority.Apr 18 2016, 10:21 PM
RobH added a comment.Apr 18 2016, 10:22 PM

All of this appears correct to me. Additionally, the partitioning and OS should be the same as what is currently running for restbase2001-2006, as these hosts have the same or similar (or one has slightly different intel ssds) disks installed.

Peachey88 added projects: SRE, ops-codfw.Apr 19 2016, 1:26 AM
Peachey88 updated the task description. (Show Details)
fgiunchedi added a comment.Apr 19 2016, 9:21 AM

thanks @Papaul, rows should be B/C/D, one in each, if possible not located in the same rack as existing restbase systems

Papaul added a comment.Apr 19 2016, 3:30 PM
server name rack location
restbase2001B5
restbase2002B8
restbase2003C1
restbase2004C5
restbase2005D1
restbase2006D5

layout option

server name rack location
restbase2007B1
restbase2008C1 or C5
restbase2009D3 or D4 or D8

@fgiunchedi if we have to rack those servers in different racks than the existing restbase servers, we will not be able to do that in row C. one of the rack (C1or C5) will have to have 2 restbase servers. Please see table above for details options. I know D3 and D4 are received for mw* servers but I may be wrong and D8 i don't know. if we have to stay away of those racks than one of the rack (D1 or D5) will have to have also 2 restbase servers. Thanks

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 3:30 PM
fgiunchedi added a comment.Apr 19 2016, 4:45 PM

ok, thanks @Papaul ! if we have to co-locate in the same rack in row C that's fine too, I'll leave it to you whether C1 or C5

Papaul added a comment.Apr 19 2016, 4:56 PM

Thanks @fgiunchedi here is the final layout

server name rack location
restbase2007B1
restbase2008C1
restbase2009D1
fgiunchedi added a comment.Apr 20 2016, 3:30 PM

excellent @Papaul, let me know when done or if you need anything from me

Papaul added a comment.Apr 20 2016, 3:34 PM

Nothing for now, since I am waiting on the other 2 servers . ETA says 4-22. thanks

Papaul updated the task description. (Show Details)Apr 20 2016, 3:39 PM
Papaul updated the task description. (Show Details)Apr 25 2016, 8:00 PM
Restricted Application added a subscriber: Southparkfan. · View Herald TranscriptApr 25 2016, 8:00 PM
Papaul updated the task description. (Show Details)Apr 25 2016, 8:26 PM
Papaul updated the task description. (Show Details)Apr 26 2016, 9:21 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Apr 26 2016, 9:23 PM
Papaul created subtask T133788: rack/setup/deploy restbase200[7-9] switch configuration .Apr 27 2016, 4:48 PM
Papaul updated the task description. (Show Details)Apr 27 2016, 4:51 PM
Papaul updated the task description. (Show Details)Apr 27 2016, 5:14 PM
RobH added a comment.Apr 27 2016, 6:27 PM

Please note that of the three new systems, only one of the three systems was ordered with SSDs.

Please make the system with the Intel SSDs restbase2009.

restbase2007-2008 need to have the samsung 850 pro ssds installed into them, as they shipped with the drive sleds, but no SSDs. The SSDs for these should be taken from spares + removing ssds from restbase200[1-6]. Please work with @fgiunchedi on which ssds to remove for this task.

RobH closed subtask T133788: rack/setup/deploy restbase200[7-9] switch configuration as Resolved.Apr 27 2016, 6:29 PM
fgiunchedi added a comment.Apr 28 2016, 9:21 AM

@Papaul those would be the ssds we installed and then removed in T127333

Papaul added a comment.Apr 28 2016, 2:46 PM

restbase2007 is the server with the Intel SSDs and it is racked in B1 and restbaset2009 is rack in D1. I will move restbase2007 in D1 and rename it restbase2009 and move restbase2009 which is in D1 to B1 and rename it restbase2007 just to keep things organized.

Papaul updated the task description. (Show Details)Apr 28 2016, 4:56 PM
Papaul reopened subtask T133788: rack/setup/deploy restbase200[7-9] switch configuration as Open.Apr 29 2016, 2:47 PM
RobH mentioned this in T133788: rack/setup/deploy restbase200[7-9] switch configuration .Apr 29 2016, 5:06 PM
RobH closed subtask T133788: rack/setup/deploy restbase200[7-9] switch configuration as Resolved.
MoritzMuehlenhoff subscribed.May 2 2016, 4:57 PM
Stashbot subscribed.May 2 2016, 5:23 PM

Mentioned in SAL [2016-05-02T17:23:49Z] <robh> restbase2004 offline for next few hours for comparison work for new systems T132976

Papaul added a comment.May 2 2016, 8:03 PM

I compared restbase2004 and restbase2008, all the cables are connected the same way. The Bios setting are also the same. the issue was with the Smart Array P440ar configuration .You need to configure the array manually.

Access the HP Samrt Storage administrator web UI to manully setup the disks in there

F9 to go to the system utilities
sytstem configuration
Embeddred RAID : Smart Array P440ar configuration
Exit and lunch HP Samrt Storage Administrator
go to Smart Array P440ar
under Astion select create Array
in the select physical drives for the new array windows, select the first drive and click create Array
in the next window leave the setting as it is and click create logical drive
do the same steps agan for each drive.
select set bootable logical drive/volume
and select the first drive as primart boot logical drive/volume

gerritbot subscribed.May 2 2016, 8:11 PM

Change 286515 had a related patch set uploaded (by Dzahn):
Revert "DHCP: changing the install to trusty to test since jessie is not detecting the disks Bug: T132976"

https://gerrit.wikimedia.org/r/286515

gerritbot added a project: Patch-For-Review.May 2 2016, 8:11 PM
gerritbot added a comment.May 2 2016, 8:15 PM

Change 286515 merged by Dzahn:
Revert "DHCP: changing the install to trusty to test since jessie is not detecting the disks Bug: T132976"

https://gerrit.wikimedia.org/r/286515

RobH mentioned this in T134197: plug in restbase2004 network cable.May 2 2016, 8:40 PM
Papaul reassigned this task from Papaul to fgiunchedi.May 2 2016, 10:37 PM
Papaul updated the task description. (Show Details)

Installation complete

fgiunchedi reassigned this task from fgiunchedi to Papaul.May 3 2016, 10:02 AM

@Papaul I'm seeing 4x SSD on restbase200[78] (the machines with 1TB samsung) though those should have 5x, there should be 1TB samsung spare from T127333 for you to install

gerritbot added a comment.May 3 2016, 2:57 PM

Change 286663 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase200[789] instances

https://gerrit.wikimedia.org/r/286663

gerritbot added a comment.May 3 2016, 2:59 PM

Change 286663 merged by Filippo Giunchedi:
cassandra: add restbase200[789] instances

https://gerrit.wikimedia.org/r/286663

gerritbot added a comment.May 3 2016, 3:20 PM

Change 286665 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase200[789]

https://gerrit.wikimedia.org/r/286665

gerritbot added a comment.May 3 2016, 3:36 PM

Change 286665 merged by Filippo Giunchedi:
cassandra: add restbase200[789]

https://gerrit.wikimedia.org/r/286665

Papaul reassigned this task from Papaul to fgiunchedi.May 3 2016, 4:11 PM

@fgiunchedi I added the disks to restbase2007 and restbase2008. Both have now 5 SSDs

Stashbot added a comment.May 3 2016, 4:58 PM

Mentioned in SAL [2016-05-03T16:58:47Z] <godog> bootstrap restbase2009-a T132976

gerritbot added a comment.May 6 2016, 8:33 AM

Change 287180 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase2009-b

https://gerrit.wikimedia.org/r/287180

gerritbot added a comment.May 6 2016, 8:35 AM

Change 287180 merged by Filippo Giunchedi:
cassandra: add restbase2009-b

https://gerrit.wikimedia.org/r/287180

Stashbot added a comment.May 9 2016, 9:15 AM

Mentioned in SAL [2016-05-09T09:15:08Z] <godog> bootstrap restbase2007-a T132976

Stashbot added a comment.May 10 2016, 8:29 AM

Mentioned in SAL [2016-05-10T08:29:38Z] <godog> bootstrap restbase2007-b T132976

Stashbot added a comment.May 10 2016, 4:04 PM

Mentioned in SAL [2016-05-10T16:04:19Z] <godog> nodetool cleanup on restbase2005 T132976

Stashbot added a comment.May 11 2016, 8:28 AM

Mentioned in SAL [2016-05-11T08:28:32Z] <godog> bootstrap restbase2008-a T132976

Stashbot added a comment.May 13 2016, 9:48 AM

Mentioned in SAL [2016-05-13T09:48:44Z] <godog> nodetool cleanup on restbase2006 T132976

gerritbot added a comment.May 16 2016, 9:24 AM

Change 288894 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase2008-b

https://gerrit.wikimedia.org/r/288894

gerritbot added a comment.May 16 2016, 9:27 AM

Change 288894 merged by Filippo Giunchedi:
cassandra: add restbase2008-b

https://gerrit.wikimedia.org/r/288894

fgiunchedi mentioned this in rOPUP3303fd90a030: cassandra: add restbase2008-b.May 16 2016, 9:31 AM
fgiunchedi mentioned this in T95253: Finish conversion to multiple Cassandra instances per hardware node.May 20 2016, 2:00 PM
fgiunchedi added a comment.May 20 2016, 2:06 PM

all instances have been bootstrapped, left to do:

  • deploy restbase on restbase200[789] if not already
  • add restbase200[789] to conftool and pool them in lvs
Eevans added a project: Cassandra.May 20 2016, 2:58 PM
Eevans moved this task from Backlog to In-Progress on the Cassandra board.
gerritbot added a comment.May 23 2016, 11:23 AM

Change 290199 had a related patch set uploaded (by Filippo Giunchedi):
restbase: add restbase200[789] to conftool-data

https://gerrit.wikimedia.org/r/290199

gerritbot added a comment.May 23 2016, 12:13 PM

Change 290199 merged by Filippo Giunchedi:
restbase: add restbase200[789] to conftool-data

https://gerrit.wikimedia.org/r/290199

fgiunchedi mentioned this in rOPUP55be493649fd: restbase: add restbase200[789] to conftool-data.May 23 2016, 12:16 PM
fgiunchedi closed this task as Resolved.May 23 2016, 12:16 PM

restbase200[789] bootstrapped each with two instances and restbase running, resolving

fgiunchedi mentioned this in rOPUPdd1c10454912: cassandra: add restbase2008-b.Jun 17 2016, 6:09 PM
fgiunchedi mentioned this in rOPUP69ce2d92d0a9: cassandra: add restbase200[789].
Dzahn mentioned this in rOPUP7a03ac1e019e: Revert "DHCP: changing the install to trusty to test since jessie is not….Jun 17 2016, 6:10 PM
Papaul mentioned this in rOPUP4e4a38c0b6fd: DHCP: changing the install to trusty to test since jessie is not detecting….Jun 17 2016, 6:10 PM
Papaul mentioned this in rOPUP38c49054e1a4: adding install params for restbase200[7-9] Bug:T132976.
Papaul mentioned this in rOPUP5ff979d21b60: adding install params for kafka200[1-2] Bug:T132976.
Papaul mentioned this in rOPUPab78698557f2: DHCP: MAC address chang, swaped restbase2007 with restbase2009 Bug:T132976.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL