Page MenuHomePhabricator
Create Task
Maniphest T166237

rack/setup/install labtestvirt2003
Closed, ResolvedPublic
Actions

Description

This task will track the racking and setup of the new labtestvirt2003 server for codfw. This was ordered on T163030

Racking Proposal: This host has to reside in the labs-hosts1-b-codfw subnet, so it has to go in row B. There are labs-hosts1 subnets reserved for the other rows, but they aren't in use yet, as we don't support the cloud/labs virt hosts across rows quite yet. So rack this in row B, in whatever rack has 1Gbps networking and the most room/space/power availability (ie: not in frack or 10G racks.)

labtestvirt2003:

  • - receive in system on procurement task T163030
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added
  • - network port setup (description, enable, vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

Please note there was confusion on hostname use, since racktables didn't show labtestvirt2002 in use, when a host already existed. Task T166598 was created to correct that. The host being setup on this task should now be named labtestvirt2003.

Details

SubjectRepoBranchLines +/-
setting labtestvirt2003 into site.ppoperations/puppetproduction+1 -1
Add partman recipe and DHCP entries for labtestvirt2003operations/puppetproduction+8 -0
DNS: Add mgmt and production DNS for labtestvirt2003operations/dnsmaster+7 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.May 24 2017, 4:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 24 2017, 4:51 PM
Paladox subscribed.May 24 2017, 5:37 PM
Papaul created subtask T166564: codfw: labtestvirt2002 swith port configuration .May 30 2017, 2:50 PM
Papaul updated the task description. (Show Details)May 30 2017, 2:56 PM
Papaul added a comment.EditedMay 30 2017, 3:08 PM

@chasemp what partman recipe do you want to use for the server? We have :

  • raid10-gpt.cfg
  • raid10-gpt-srv-ext4.cfg
  • raid10-gpt-srv-lvm-ext4.cfg
Papaul added a comment.May 30 2017, 5:35 PM

@RobH I have already mgmt and productions DNS for labtestvirt2002 shouldn't this be labtestvirt2003

RobH added a comment.May 30 2017, 5:53 PM

It seems that there is confusion, due to the fact that racktables shows two labtestvirt2001 systems.

I went ahead and connected to the mgmt dns for the existing labtestvirt2002, and it points to this system: https://racktables.wikimedia.org/index.php?page=object&object_id=1317

WMF3810 is labtestvirt2002. (There will be a new task to correct that one shortly.)

The system being setup on this task T166237, should be renamed to labtestvirt2003, since the other system already exists, and is online as labtestvirt2002.

RobH renamed this task from rack/setup/install labtestvirt2002 to rack/setup/install labtestvirt2003.May 30 2017, 5:55 PM
RobH mentioned this in T166598: correct label on labtestvirt2002/WMF3810.
RobH created subtask T166598: correct label on labtestvirt2002/WMF3810.
RobH updated the task description. (Show Details)
gerritbot subscribed.May 30 2017, 9:49 PM

Change 356302 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for labtestvirt2003

https://gerrit.wikimedia.org/r/356302

gerritbot added a project: Patch-For-Review.May 30 2017, 9:49 PM
gerritbot added a comment.May 30 2017, 11:00 PM

Change 356302 merged by Dzahn:
[operations/dns@master] DNS: Add mgmt and production DNS for labtestvirt2003

https://gerrit.wikimedia.org/r/356302

Papaul closed subtask T166598: correct label on labtestvirt2002/WMF3810 as Resolved.May 31 2017, 4:43 PM
Papaul updated the task description. (Show Details)May 31 2017, 4:55 PM
Papaul added a comment.May 31 2017, 4:58 PM

@chasemp the others lab servers in the DHCP file are pointing to the Trusty install do you want to install Trusty on labtestvirt2003 or Jessie ?

• chasemp added a comment.Jun 1 2017, 12:30 PM
In T166237#3304793, @Papaul wrote:

@chasemp the others lab servers in the DHCP file are pointing to the Trusty install do you want to install Trusty on labtestvirt2003 or Jessie ?

@Papaul, yes the openstack infrastructure is still on Trusty atm thanks

• chasemp added a subscriber: Andrew.Jun 1 2017, 12:40 PM
In T166237#3301373, @RobH wrote:

It seems that there is confusion, due to the fact that racktables shows two labtestvirt2001 systems.

I went ahead and connected to the mgmt dns for the existing labtestvirt2002, and it points to this system: https://racktables.wikimedia.org/index.php?page=object&object_id=1317

WMF3810 is labtestvirt2002. (There will be a new task to correct that one shortly.)

The system being setup on this task T166237, should be renamed to labtestvirt2003, since the other system already exists, and is online as labtestvirt2002.

thanks @RobH, sorry for the confusion. At some point I think @Andrew renamed while we debugged the scheduler and we missed racktables.

• chasemp added a comment.Jun 1 2017, 3:18 PM
In T166237#3300388, @Papaul wrote:

@chasemp what partman recipe do you want to use for the server? We have :

raid10-gpt-srv-lvm-ext4.cfg

seems fine, thanks papaul

Papaul added a comment.Jun 1 2017, 3:27 PM

@chasemp thanks.

gerritbot added a comment.Jun 1 2017, 3:45 PM

Change 356617 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Add partman recipe and DHCP entries for labtestvirt2003

https://gerrit.wikimedia.org/r/356617

gerritbot added a comment.Jun 1 2017, 3:55 PM

Change 356617 merged by RobH:
[operations/puppet@production] Add partman recipe and DHCP entries for labtestvirt2003

https://gerrit.wikimedia.org/r/356617

RobH updated the task description. (Show Details)Jun 1 2017, 3:56 PM
RobH claimed this task.Jun 1 2017, 4:03 PM
RobH added a subscriber: Papaul.
RobH closed subtask T166564: codfw: labtestvirt2002 swith port configuration as Resolved.Jun 1 2017, 4:06 PM
gerritbot added a comment.Jun 1 2017, 4:32 PM

Change 356625 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting labtestvirt2003 into site.pp

https://gerrit.wikimedia.org/r/356625

gerritbot added a comment.Jun 1 2017, 4:39 PM

Change 356625 merged by RobH:
[operations/puppet@production] setting labtestvirt2003 into site.pp

https://gerrit.wikimedia.org/r/356625

RobH updated the task description. (Show Details)Jun 1 2017, 4:42 PM
RobH added a comment.Jun 1 2017, 4:45 PM

Tried to add into site.pp but seems there is an error condition due to kernels in use (system was just freshly installed.)

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: nova-compute not installed on buggy kernels. On 3.13 series kernels, instance suspension causes complete system lockup. Try installing linux-image-generic-lts-xenial at /etc/puppet/modules/openstack/manifests/nova/compute.pp:123 on node labtestvirt2003.codfw.wmnet
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

RobH reassigned this task from RobH to • chasemp.Jun 1 2017, 5:12 PM

So it kicked back that failure, and though it now has the puppet keys and such installed, has failures left over from that failed install.

@chasemp: if you guys want to re-enable the site.pp settings and debug, I'm kickign this over to you. If you prefer I reinstall it again, just kick it back to me!

Andrew added a comment.Jun 1 2017, 11:00 PM

Labtestvirt2003 is installed now, and properly attached to rabbitmq and the nova controller.

  • It is not currently in the scheduler pool (but easy to add)
  • There's one issue with the prometheus setup, which is T166843
Andrew closed this task as Resolved.Jun 1 2017, 11:04 PM
Andrew updated the task description. (Show Details)
• chasemp mentioned this in T183035: labtestvirt2003 does not survive reboot on normal labvirt kernel of 4.4.0-81-generic.Dec 15 2017, 9:08 PM
• chasemp mentioned this in T183167: Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric.Dec 18 2017, 6:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL