@Jgreen these have been received is this urgent or could I wait till after fundraising to rack and cable these?

@Jclark-ctr Sorry for the delay on this. They weren't urgent and now the December fundraising is complete. You are clear to rack and cable these hosts when you are available.

frbast1002 port 9
frmon1002 port 11
frpig1002 port 1

Change 894044 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/dns@master] Add hosts frbast1002,frmon1002,frpig1002, remove frauth1001,frpm1001

https://gerrit.wikimedia.org/r/894044

Change 894044 merged by Jgreen:

[operations/dns@master] Add hosts frbast1002,frmon1002,frpig1002, remove frauth1001,frpm1001

https://gerrit.wikimedia.org/r/894044

I received an idrac error on 3 of these hosts, I confirmed with Jeff that he is not able to access the host. I am going to try and update the idrac firmware

@Cmjohnson @Jgreen i did a quick look in Netbox for frbast1002 mgmt IP address it looks like this node is using 10.64.40.36/26 or eqiad mgmt network is 10.64.40.192/26 so no ip address below 192 maybe this is why your guys are having issue because the IP address is not in the management network. I think this will be also true for the other 2 servers.
https://netbox.wikimedia.org/ipam/prefixes/118/

Updated netbox and idracs on all three servers

frbast1002: vlan:frack-bastion-eqiad ip:10.64.40.196
frmon1002: vlan:frack-administration-eqiad ip:10.64.40.198
frpig1002: vlan:frack-listener-eqiad ip:10.64.40.202

@Jgreen can you please confirm that you an not access those servers so you can take over the task?

thanks

In T319460#8782417, @Papaul wrote:

@Jgreen can you please confirm that you an not access those servers so you can take over the task?

thanks

I can access them now, thanks! Is there still firmware updating to be done? Otherwise I think we're all set.

@Jgreen I can check and let you know on the firmware update.

@Jgreen we will have to update the firmware on those.

I just attempted to build frbast1002 and frpig1002 and neither got a dhcp offer. Could we please verify that all the hosts are in the correct VLANs? Thanks.

@Dwisehaupt i checked on the switch all the interfaces are configured and up maybe the server were not added to DNS since we do not manage Frack DNS in netbox we need to add those servers manually into the DNS file.l I will cehck the DNS file once i am home. Thanks

papaul@fasw-c-eqiad> show interfaces ge-[0-1]/0/9 descriptions 
Interface       Admin Link Description
ge-0/0/9        up    up   frbast1002
ge-1/0/9        up    up   frbast1002

{master:0}
papaul@fasw-c-eqiad> show interfaces ge-[0-1]/0/11 descriptions   
Interface       Admin Link Description
ge-0/0/11       up    up   frmon1002
ge-1/0/11       up    up   frmon1002

{master:0}
papaul@fasw-c-eqiad> show interfaces ge-[0-1]/0/1 descriptions     
Interface       Admin Link Description
ge-0/0/1        up    up   frpig1002
ge-1/0/1        up    up   frpig1002

@Papaul Thanks, I have verified they are in DNS.

I think there may be some crossing in cables or vlans. When I try to build a host, I'm seeing the entry in the dhcp server logs, but they are coming from the wrong subnets.

When trying to build frbast1002, I see the following:

    frbast1002:
        ipaddress: '10.64.40.36'
        macaddress: 'F4:EE:08:3F:01:FF'

$ host frbast1002
frbast1002.frack.eqiad.wmnet has address 10.64.40.36

$ tail -f /var/log/syslog | grep dhcp
Apr 19 23:11:20 frpm1002 dhcpd[4010214]: DHCPDISCOVER from f4:ee:08:3f:01:ff via 10.64.40.161: network 10.64.40.160/27: no free leases
Apr 19 23:11:24 frpm1002 dhcpd[4010214]: DHCPDISCOVER from f4:ee:08:3f:01:ff via 10.64.40.161: network 10.64.40.160/27: no free leases

When trying to build frpig1002, I see:

    frpig1002:
        ipaddress: '10.64.40.165'
        macaddress: 'f4:ee:08:3f:04:c7'

$ host frpig1002
frpig1002.frack.eqiad.wmnet has address 10.64.40.165

$ tail -f /var/log/syslog | grep dhcp
Apr 19 23:25:52 frpm1002 dhcpd[4010214]: DHCPDISCOVER from f4:ee:08:3f:04:c7 via 10.64.40.33: network 10.64.40.32/27: no free leases
Apr 19 23:25:56 frpm1002 dhcpd[4010214]: DHCPDISCOVER from f4:ee:08:3f:04:c7 via 10.64.40.33: network 10.64.40.32/27: no free leases

Thanks for any help.

@Cmjohnson I did some debugging with @Papaul online and we think there may be a crossed cable issue between frbast1002 and frpig1002.

Could you please verify the cable paths of the two hosts when you are next on site? Thanks.

Verified cables and corrected them @Dwisehaupt

@Jclark-ctr Thanks! I can verify that worked and I have been able to start building out the hosts.

Change 912340 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/dns@master] Add mappings for new frbast and payments-listener hosts

https://gerrit.wikimedia.org/r/912340

Change 912340 merged by Jgreen:

[operations/dns@master] Add mappings for new frbast and payments-listener hosts

https://gerrit.wikimedia.org/r/912340

NAT mappings in place. Tested frbast1002 usage via a host file change on my local host with my current config and the stock config we provide users.

Change 915811 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/dns@master] Direct frbast.wm.o at the new frbast1002 host

https://gerrit.wikimedia.org/r/915811

Change 915811 merged by Jgreen:

[operations/dns@master] Direct frbast.wm.o at the new frbast1002 host

https://gerrit.wikimedia.org/r/915811

Change 916617 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] Add monitoring for new fr-tech hosts

https://gerrit.wikimedia.org/r/916617

Change 916617 merged by Jgreen:

[operations/puppet@production] Add monitoring for new fr-tech hosts

https://gerrit.wikimedia.org/r/916617

Change 920777 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/dns@master] Shift frbast names to using the new hosts

https://gerrit.wikimedia.org/r/920777

Change 920779 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/dns@master] Shift frbast names to using the new hosts

https://gerrit.wikimedia.org/r/920779

Change 920777 abandoned by Dwisehaupt:

[operations/dns@master] Shift frbast names to using the new hosts

Reason:

https://gerrit.wikimedia.org/r/920777

Change 920779 merged by Jgreen:

[operations/dns@master] Shift frbast names to using the new hosts

https://gerrit.wikimedia.org/r/920779

@Dwisehaupt anything else we can do to help with this task?

@Papaul Your side is all set. We have some switch overs scheduled for the end of the month to finish up our side of the task too. Thanks for checking in.

Change 931985 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/dns@master] Switch payments-listener-eqiad to the new server/ip.

https://gerrit.wikimedia.org/r/931985

Change 931985 merged by Jgreen:

[operations/dns@master] Switch payments-listener-eqiad to the new server/ip.

https://gerrit.wikimedia.org/r/931985

Confirmed traffic to frpig1001 has stopped. Taking it out of monitoring so we can shut it down and start the decom.

Change 932257 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] Remove frpig1001.frack.eqiad.wmnet from nsca_frack.cfg.erb in prep for decommission.

https://gerrit.wikimedia.org/r/932257

Change 932257 merged by Filippo Giunchedi:

[operations/puppet@production] Remove frpig1001 from nsca_frack.cfg.erb in prep for decom.

https://gerrit.wikimedia.org/r/932257

Change 932420 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] Switch blackbox_smoke_hosts check from frpig.* to payments-listener-.*

https://gerrit.wikimedia.org/r/932420

Change 932420 merged by Filippo Giunchedi:

[operations/puppet@production] Switch blackbox_smoke_hosts check from frpig.* to payments-listener-.*

https://gerrit.wikimedia.org/r/932420

Change 933199 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] Add frmon1002 to monitoring

https://gerrit.wikimedia.org/r/933199

Change 933199 merged by Jgreen:

[operations/puppet@production] Add frmon1002 to monitoring

https://gerrit.wikimedia.org/r/933199