Page MenuHomePhabricator

rack/setup/deploy restbase200[7-9]
Closed, ResolvedPublic

Description

The systems for this task have been ordered on procurement task T130218. Once these arrive on-site, they will be racked in A5, C5 and D5

restbase2007

  • receive in normally via T130218
  • rack in b1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

restbase2008

  • receive in normally via T130218
  • rack in c1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

restbase2009

  • receive in normally via T130218
  • rack in d1-codfw
  • add mgmt dns entries for both asset tag and hostname
  • add production dns entries (private vlan)
  • create sub-task with network port info for setup
  • update dhcp file with MAC address
  • update install_server module (use cassandrahosts-4ssd-srv.cfg)
  • install OS - Jessie
  • service implementation (hand off to @Filippo)

Event Timeline

Papaul created this task.Apr 18 2016, 10:18 PM

@fgiunchedi pleae confirm that the setup and racking information for the new restbase servers are correct. Thanks

Papaul triaged this task as High priority.Apr 18 2016, 10:21 PM
RobH added a comment.Apr 18 2016, 10:22 PM

All of this appears correct to me. Additionally, the partitioning and OS should be the same as what is currently running for restbase2001-2006, as these hosts have the same or similar (or one has slightly different intel ssds) disks installed.

Peachey88 updated the task description. (Show Details)

thanks @Papaul, rows should be B/C/D, one in each, if possible not located in the same rack as existing restbase systems

server name rack location
restbase2001B5
restbase2002B8
restbase2003C1
restbase2004C5
restbase2005D1
restbase2006D5

layout option

server name rack location
restbase2007B1
restbase2008C1 or C5
restbase2009D3 or D4 or D8

@fgiunchedi if we have to rack those servers in different racks than the existing restbase servers, we will not be able to do that in row C. one of the rack (C1or C5) will have to have 2 restbase servers. Please see table above for details options. I know D3 and D4 are received for mw* servers but I may be wrong and D8 i don't know. if we have to stay away of those racks than one of the rack (D1 or D5) will have to have also 2 restbase servers. Thanks

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 3:30 PM

ok, thanks @Papaul ! if we have to co-locate in the same rack in row C that's fine too, I'll leave it to you whether C1 or C5

Thanks @fgiunchedi here is the final layout

server name rack location
restbase2007B1
restbase2008C1
restbase2009D1

excellent @Papaul, let me know when done or if you need anything from me

Nothing for now, since I am waiting on the other 2 servers . ETA says 4-22. thanks

Papaul updated the task description. (Show Details)Apr 20 2016, 3:39 PM
Papaul updated the task description. (Show Details)Apr 25 2016, 8:00 PM
Restricted Application added a subscriber: Southparkfan. · View Herald TranscriptApr 25 2016, 8:00 PM
Papaul updated the task description. (Show Details)Apr 25 2016, 8:26 PM
Papaul updated the task description. (Show Details)Apr 26 2016, 9:21 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Apr 26 2016, 9:23 PM
Papaul updated the task description. (Show Details)Apr 27 2016, 4:51 PM
Papaul updated the task description. (Show Details)Apr 27 2016, 5:14 PM
RobH added a comment.Apr 27 2016, 6:27 PM

Please note that of the three new systems, only one of the three systems was ordered with SSDs.

Please make the system with the Intel SSDs restbase2009.

restbase2007-2008 need to have the samsung 850 pro ssds installed into them, as they shipped with the drive sleds, but no SSDs. The SSDs for these should be taken from spares + removing ssds from restbase200[1-6]. Please work with @fgiunchedi on which ssds to remove for this task.

@Papaul those would be the ssds we installed and then removed in T127333

restbase2007 is the server with the Intel SSDs and it is racked in B1 and restbaset2009 is rack in D1. I will move restbase2007 in D1 and rename it restbase2009 and move restbase2009 which is in D1 to B1 and rename it restbase2007 just to keep things organized.

Papaul updated the task description. (Show Details)Apr 28 2016, 4:56 PM

Mentioned in SAL [2016-05-02T17:23:49Z] <robh> restbase2004 offline for next few hours for comparison work for new systems T132976

Papaul added a comment.May 2 2016, 8:03 PM

I compared restbase2004 and restbase2008, all the cables are connected the same way. The Bios setting are also the same. the issue was with the Smart Array P440ar configuration .You need to configure the array manually.

Access the HP Samrt Storage administrator web UI to manully setup the disks in there

F9 to go to the system utilities
sytstem configuration
Embeddred RAID : Smart Array P440ar configuration
Exit and lunch HP Samrt Storage Administrator
go to Smart Array P440ar
under Astion select create Array
in the select physical drives for the new array windows, select the first drive and click create Array
in the next window leave the setting as it is and click create logical drive
do the same steps agan for each drive.
select set bootable logical drive/volume
and select the first drive as primart boot logical drive/volume

Change 286515 had a related patch set uploaded (by Dzahn):
Revert "DHCP: changing the install to trusty to test since jessie is not detecting the disks Bug: T132976"

https://gerrit.wikimedia.org/r/286515

Change 286515 merged by Dzahn:
Revert "DHCP: changing the install to trusty to test since jessie is not detecting the disks Bug: T132976"

https://gerrit.wikimedia.org/r/286515

Papaul updated the task description. (Show Details)May 2 2016, 10:37 PM
Papaul reassigned this task from Papaul to fgiunchedi.

Installation complete

fgiunchedi reassigned this task from fgiunchedi to Papaul.May 3 2016, 10:02 AM

@Papaul I'm seeing 4x SSD on restbase200[78] (the machines with 1TB samsung) though those should have 5x, there should be 1TB samsung spare from T127333 for you to install

Change 286663 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase200[789] instances

https://gerrit.wikimedia.org/r/286663

Change 286663 merged by Filippo Giunchedi:
cassandra: add restbase200[789] instances

https://gerrit.wikimedia.org/r/286663

Change 286665 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase200[789]

https://gerrit.wikimedia.org/r/286665

Change 286665 merged by Filippo Giunchedi:
cassandra: add restbase200[789]

https://gerrit.wikimedia.org/r/286665

Papaul reassigned this task from Papaul to fgiunchedi.May 3 2016, 4:11 PM

@fgiunchedi I added the disks to restbase2007 and restbase2008. Both have now 5 SSDs

Mentioned in SAL [2016-05-03T16:58:47Z] <godog> bootstrap restbase2009-a T132976

Change 287180 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase2009-b

https://gerrit.wikimedia.org/r/287180

Change 287180 merged by Filippo Giunchedi:
cassandra: add restbase2009-b

https://gerrit.wikimedia.org/r/287180

Mentioned in SAL [2016-05-09T09:15:08Z] <godog> bootstrap restbase2007-a T132976

Mentioned in SAL [2016-05-10T08:29:38Z] <godog> bootstrap restbase2007-b T132976

Mentioned in SAL [2016-05-10T16:04:19Z] <godog> nodetool cleanup on restbase2005 T132976

Mentioned in SAL [2016-05-11T08:28:32Z] <godog> bootstrap restbase2008-a T132976

Mentioned in SAL [2016-05-13T09:48:44Z] <godog> nodetool cleanup on restbase2006 T132976

Change 288894 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase2008-b

https://gerrit.wikimedia.org/r/288894

Change 288894 merged by Filippo Giunchedi:
cassandra: add restbase2008-b

https://gerrit.wikimedia.org/r/288894

all instances have been bootstrapped, left to do:

  • deploy restbase on restbase200[789] if not already
  • add restbase200[789] to conftool and pool them in lvs
Eevans moved this task from Backlog to In-Progress on the Cassandra board.

Change 290199 had a related patch set uploaded (by Filippo Giunchedi):
restbase: add restbase200[789] to conftool-data

https://gerrit.wikimedia.org/r/290199

Change 290199 merged by Filippo Giunchedi:
restbase: add restbase200[789] to conftool-data

https://gerrit.wikimedia.org/r/290199

fgiunchedi closed this task as Resolved.May 23 2016, 12:16 PM

restbase200[789] bootstrapped each with two instances and restbase running, resolving