Page MenuHomePhabricator
Create Task
Maniphest T167157

rack/setup/install labtestpuppetmaster2001
Closed, ResolvedPublic
Actions

Description

This task will track the racking and setup/installation of the new labstestpuppetmaster2001 system in codfw.

hostname label: labtestpuppetmaster1001 likely wont fit on the name label. try labtestpm2001 and have the visible label in racktables note the difference.

Racking Plan: This host will be placed in the public vlan, so it can be racked in any 1GBE rack in any row in codfw.

  • - receive in system on procurement task T164517
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added https://gerrit.wikimedia.org/r/#/c/358485/
  • - network port setup (description, enable, vlan) (create sub task if not done immediately)
  • - operations/puppet update (install_server at minimum, other files if possible) - raid1,lvm,srv,ext4.
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
Add AAAA for labtestpuppetmaster2001operations/dnsmaster+3 -0
fix DNS for labtestpuppetmaster, 1002 != 2001operations/dnsmaster+1 -1
setting labtestpuppetmaster2001 production dnsoperations/dnsmaster+4 -4
DNS: Add mgmt and production DNS for labtestpuppetmaster2001operations/dnsmaster+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jun 6 2017, 4:58 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 6 2017, 4:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul updated the task description. (Show Details)Jun 6 2017, 8:08 PM
Papaul updated the task description. (Show Details)Jun 7 2017, 4:10 PM
Papaul created subtask T167321: codfw: labtestpuppetmaster2001 switch port configuration .Jun 7 2017, 5:01 PM
gerritbot subscribed.Jun 8 2017, 3:52 PM

Change 357841 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for labtestpuppetmaster2001

https://gerrit.wikimedia.org/r/357841

gerritbot added a project: Patch-For-Review.Jun 8 2017, 3:52 PM
gerritbot added a comment.Jun 8 2017, 4:02 PM

Change 357841 merged by RobH:
[operations/dns@master] DNS: Add mgmt and production DNS for labtestpuppetmaster2001

https://gerrit.wikimedia.org/r/357841

RobH updated the task description. (Show Details)Jun 8 2017, 4:09 PM
gerritbot added a comment.Jun 12 2017, 9:22 PM

Change 358485 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] setting labtestpuppetmaster2001 production dns

https://gerrit.wikimedia.org/r/358485

gerritbot added a comment.Jun 12 2017, 9:23 PM

Change 358485 merged by RobH:
[operations/dns@master] setting labtestpuppetmaster2001 production dns

https://gerrit.wikimedia.org/r/358485

RobH removed a project: Patch-For-Review.Jun 12 2017, 9:38 PM
RobH updated the task description. (Show Details)
RobH closed subtask T167321: codfw: labtestpuppetmaster2001 switch port configuration as Resolved.
Papaul reopened subtask T167321: codfw: labtestpuppetmaster2001 switch port configuration as Open.Jun 13 2017, 1:51 AM
RobH mentioned this in T164515: codfw: (1) labtest puppetmaster.Jun 14 2017, 6:03 PM
Papaul added a comment.Jun 16 2017, 1:16 PM

I troubleshoot this with Daniel by replacing replacing eth0 with eth1 MAC address in the DHCP file but same problem can not boot also from eth1. I will run a hardware diagnostic on th system.

bd808 moved this task from Triage to Backlog on the Cloud-Services board.Jun 17 2017, 8:02 PM
Papaul added a comment.EditedJun 20 2017, 5:22 PM

In the process of troubleshooting the pxe boot issue on this system, I setup a test dhcp/dns/tftp server on my laptop and boot the server to it the and it works with no problem so the NIC 1 works with no problem. See below for screen chat. I reconnected the server back to WMF network and try to pxe boot again and i am getting this in the dhcp log :

DHCPDISCOVER from 30:e1:71:63:5e:5c via 208.80.153.98: network 208.80.153.96/27: no free leases
Jun 20 17:21:43 install2002 dhcpd[11106]: DHCPDISCOVER from 30:e1:71:63:5e:5c via 208.80.153.99: network 208.80.153.96/27: no free leases
This tells me that something is wrong with DNS

Screen chat
Broadcom UNDI PXE-2.1 v17.2.1
Copyright (C) 2000-2015 Broadcom Corporation
Copyright (C) 1997-2000 Intel Corporation
All rights reserved.

CLIENT MAC ADDR: 30 E1 71 63 5E 5C GUID: 32353537-3835-584D-5137-323230333232

CLIENT IP: 10.0.0.3  MASK: 255.255.255.0  DHCP IP: 10.0.0.1

GATEWAY IP: 10.0.0.1

                                                            TFTP.

PXE-T01: File not found

PXE-E3B: TFTP Error - File Not found

PXE-M0F: Exiting Broadcom PXE ROM.

gerritbot added a comment.Jun 20 2017, 5:30 PM

Change 360388 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] fix DNS for labtestpuppetmaster, 1002 != 2001

https://gerrit.wikimedia.org/r/360388

gerritbot added a project: Patch-For-Review.Jun 20 2017, 5:30 PM
gerritbot added a comment.Jun 20 2017, 5:34 PM

Change 360388 merged by Dzahn:
[operations/dns@master] fix DNS for labtestpuppetmaster, 1002 != 2001

https://gerrit.wikimedia.org/r/360388

Dzahn subscribed.Jun 20 2017, 5:37 PM
In T167157#3364338, @Papaul wrote:

Jun 20 17:21:43 install2002 dhcpd[11106]: DHCPDISCOVER from 30:e1:71:63:5e:5c via 208.80.153.99: network 208.80.153.96/27: no free leases
This tells me that something is wrong with DNS

Yes, there was something wrong with DNS, see fix above. It should work now (maybe up to 1 hour waiting because of TTL)

Papaul added a comment.Jun 20 2017, 5:37 PM

Daniel find out that for 208.80.153.108
reverse lookup = 2001
and forward lookup = 1002

He fixed it and will try install again

Papaul updated the task description. (Show Details)Jun 20 2017, 8:07 PM
Papaul updated the task description. (Show Details)Jun 20 2017, 8:57 PM
Stashbot subscribed.Jun 20 2017, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2017-06-20T21:13:21Z] <mutante> labtestpuppetmaster2001 - install-console, activate puppet, sign cert, initial puppet run, add salt key (T167157)

Papaul updated the task description. (Show Details)Jun 20 2017, 9:21 PM
Papaul added a comment.Jun 20 2017, 9:26 PM

@Andrew this is complete you can take over from here.

Thanks.

Papaul reassigned this task from Papaul to Andrew.Jun 22 2017, 2:55 PM
Papaul mentioned this in T167321: codfw: labtestpuppetmaster2001 switch port configuration .Jul 11 2017, 4:13 PM
RobH closed subtask T167321: codfw: labtestpuppetmaster2001 switch port configuration as Resolved.Jul 11 2017, 4:14 PM
gerritbot added a comment.Jul 13 2017, 12:28 AM

Change 364798 had a related patch set uploaded (by Dzahn; owner: Andrew Bogott):
[operations/dns@master] Add AAAA for labtestpuppetmaster2001

https://gerrit.wikimedia.org/r/364798

gerritbot removed a project: Cloud-Services.Jul 13 2017, 12:28 AM

Change 364798 merged by Dzahn:
[operations/dns@master] Add AAAA for labtestpuppetmaster2001

https://gerrit.wikimedia.org/r/364798

Dzahn added a comment.Jul 13 2017, 12:56 AM

It has IPv6 and reverse now.

Stashbot added a comment.Jul 13 2017, 2:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-07-13T02:26:23Z] <mutante> labtestpuppetmaster2001 - flapping icinga alerts about salt-minion starting and stopping constantly - there is an accepted salt-key but it was rejected by the master, server was reinstalled but still old key - deleted old key, accepted new key (T167157)

Dzahn added a comment.EditedJul 13 2017, 2:30 AM

root@labtestpuppetmaster2001:~# ip a s | grep inet6
inet6 2620:0:860:4:208:80:153:108/64 scope global mngtmpaddr dynamic

host labtestpuppetmaster2001.wikimedia.org
labtestpuppetmaster2001.wikimedia.org has address 208.80.153.108
labtestpuppetmaster2001.wikimedia.org has IPv6 address 2620:0:860:4:208:80:153:108

host 2620:0:860:4:208:80:153:108
8.0.1.0.3.5.1.0.0.8.0.0.8.0.2.0.4.0.0.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer labtestpuppetmaster2001.wikimedia.org.

RobH removed a project: ops-codfw.Jul 26 2017, 5:55 PM
Andrew closed this task as Resolved.Aug 7 2017, 2:26 PM

This is up and working.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits