Page MenuHomePhabricator

rack/setup/install labvirt101[5-8]
Closed, ResolvedPublic

Description

This task will track the receiving in, racking, and setup/installation of the 4 new labvirt systems ordered on procurement task T155075.

Racking plan: All the current labvirt hosts reside in row B. This was done in the past due to networking limitations within our labs infrastructure. Unless this has changed, all of these new hosts have to be racked in row B as well, so they can exist in the labs-hosts1-b-eqiad vlan. Otherwise, rack within row B where you best can distribute the new hosts across racks for some redundancy if possible.

One of the cloud services team would likely know the answer about racking above: @chasemp @bd808?

labvirt1015:

  • - receive in system on procurement T155075
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added (labs-hosts1-b-eqiad vlan)
  • - network port setup (description, enable, labs-hosts1-b-eqiad vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - wire up eth1, label port on switch, escalate to netops for setup of port/trunking.
  • - handoff for service implementation

labvirt1016:

  • - receive in system on procurement T155075
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added (labs-hosts1-b-eqiad vlan)
  • - network port setup (description, enable, labs-hosts1-b-eqiad vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - wire up eth1, label port on switch, escalate to netops for setup of port/trunking.
  • - handoff for service implementation

labvirt1017:

  • - receive in system on procurement T155075
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added (labs-hosts1-b-eqiad vlan)
  • - network port setup (description, enable, labs-hosts1-b-eqiad vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - wire up eth1, label port on switch, escalate to netops for setup of port/trunking.
  • - handoff for service implementation

labvirt1018:

  • - receive in system on procurement T155075
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added (labs-hosts1-b-eqiad vlan)
  • - network port setup (description, enable, labs-hosts1-b-eqiad vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - wire up eth1, label port on switch, escalate to netops for setup of port/trunking.
  • - handoff for service implementation

Event Timeline

RobH created this object in space Restricted Space.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH shifted this object from the Restricted Space space to the S1 Public space.May 16 2017, 10:32 PM
RobH removed a project: procurement.

Change 355792 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns entries for labvirt101[5-8] T165531

https://gerrit.wikimedia.org/r/355792

Change 355792 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns entries for labvirt101[5-8] T165531

https://gerrit.wikimedia.org/r/355792

@Cmjohnson @RobH the row b requirement for labvirts and labnets is unfortunately still real as of now. We are working on it though.

Change 357860 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357860

Change 357860 merged by Cmjohnson:
[operations/puppet@production] Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357860

Change 357870 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for several new servers, wtp1025-48, ganeti1005-1008, kubestage1001/1002, dumpsdata1001/2, labvirt1015-18 T165173 T166264 T165531 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357870

Change 357870 merged by Cmjohnson:
[operations/dns@master] Adding production dns for several new servers, wtp1025-48, ganeti1005-1008, kubestage1001/1002, dumpsdata1001/2, labvirt1015-18 and stat1005/6 T165366 T165368 T165173 T166264 T165531 T165520 T162216 T166076

https://gerrit.wikimedia.org/r/357870

mac addresses were added to dhcpd file, not sure if h/w raid is needed..i believe these came with a controller. Also @mark was looking into the switch config for me. labvirt1015 and a1016 are not in correct vlan.

Change 357879 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] Revert "Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076"

https://gerrit.wikimedia.org/r/357879

Change 357879 abandoned by RobH:
Revert "Adding mac addresses to dhcpd file for several systems, wtp1025-1046, stat1005-1006, ganeti1005-1008, labvirt1015-1018, dumpsdata1001-1002, kubestage1001-1002, analytics1069 task #'s T165173 T165366 T166264 T165531 T165368 T165520 T162216 T166076"

https://gerrit.wikimedia.org/r/357879

not sure if h/w raid is needed

Yes please!

Most of the existing labvirts have two spinny drives which are paired in a raid 1 for the OS, and then all the SSDs are in one giant raid10. If these are like that then the same plan will work.

If these boxes just have SSDs then I think we just want two raid 10 volumes, one about 100 Gb (or 80, or whatever the standard OS drive size is) and the rest with everything else. Ideally we'll be able to use the same partman recipe for these as for e.g. labvirt1014.

Thank you!

@Andrew: These hosts were reviewed and approved for order with 10 * 1.6TB Intel S3510 SSDs.

With hardware raid, you raid the ENTIRE disk, so if you want your OS data on a different raid partition than the data, it has to be split into, at minimum, 2 of the 10 1.6TB SSDs. That would lose a substantial amount of data to just silo the OS to its own hardware raid.

I'd advise that we set ALL 10 SSDs into a single raid10 array. Then we create a partman recipe that creates a LVM of the entire raid10, then a small / partition of minimal needs (120GB or so), and then uses 95% of the remainder for /srv in ext4. This seems to leverage the hardware raid, while still sticking to the data structure of existing nodes, right?

Would that be acceptable?

Would that be acceptable?

Yep, sounds great. Thank you.

Ok, further updates.

I'll write the partman recipe and get the OS isntallation done on these. However, all of these hosts will need their eth1 interface wired and it seems they are then setup in the labs-instances1-b-eqiad vlan; but @chasemp mentions they should be trunked to the other instances. (There is a good chance that Chris will just have to list/apply descriptions to the ports he connects, and then escalate the actual configuration to netops

I'll add in the check-boxes for each of these hosts in the task summary. The adding of the eth1 links do NOT block the OS installation.

@Cmjohnson @RobH thanks guys, post install assign to me and I'll take care of it.

Change 360391 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] labvirt101[5-8] install params

https://gerrit.wikimedia.org/r/360391

Change 360391 merged by RobH:
[operations/puppet@production] labvirt101[5-8] install params

https://gerrit.wikimedia.org/r/360391

RobH lowered the priority of this task from High to Medium.Jun 20 2017, 11:08 PM
RobH updated the task description. (Show Details)

Chris:

Please wire up eth1 on these systems and label their ports on the switch. Then you or I can take a look and try to duplicate the settings from an existing labvirt.

Change 360874 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt entries for labpuppetmasters T165531

https://gerrit.wikimedia.org/r/360874

Change 360874 merged by Cmjohnson:
[operations/dns@master] Adding mgmt entries for labpuppetmasters T165531

https://gerrit.wikimedia.org/r/360874

2nd ethernet connection...not setup on switch yet

Labvirt1015 2/0/21
Labvirt1016 3/0/12
labvirt1017. 7/0/11
Labvirt1018 8/0/13

Cmjohnson added a subscriber: faidon.

The second ethernet ports are cabled, cleared of the current vlan as far as I can tell. They need to be added to lab-instances. @faidon maybe can help with that.

I'll try to take care of this in the am mon or tue

[edit interfaces interface-range labs-instance-ports]
     member ge-5/0/3 { ... }
+    member ge-2/0/21;
+    member ge-4/0/12;
+    member ge-7/0/11;
+    member ge-8/0/13;

ready for initial setup :)

Change 364213 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Node defs for new labvirts: 1015-1018

https://gerrit.wikimedia.org/r/364213

Change 364213 merged by Andrew Bogott:
[operations/puppet@production] Node defs for new labvirts: 1015-1018

https://gerrit.wikimedia.org/r/364213

These are up and puppetized and running VMs.