Change 698177 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Set the partitioning scheme to new pc*

https://gerrit.wikimedia.org/r/698177

Change 698177 merged by Marostegui:

[operations/puppet@production] install_server: Set the partitioning scheme to new pc*

https://gerrit.wikimedia.org/r/698177

Change 699424 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add new parsercache hosts as insetup

https://gerrit.wikimedia.org/r/699424

Change 699424 merged by Marostegui:

[operations/puppet@production] site.pp: Add new parsercache hosts as insetup

https://gerrit.wikimedia.org/r/699424

Hi. Do we have an idea for when these hosts could be available? We have ongoing issues with parsercache (see T282761) that we hope moving to the new HW will partially mitigate.

Received host added to netbox

pc1011 A1 u21 port 9 Cableid#3963
pc1012 B1 u28 port17 Cableid#3947
pc1013 C5 u26 port24 Cableid#3410
pc1014 D6 u36 port35 Cableid#23000064

pc1011 A1 u21 port 9 Cableid#3963 IP 10.65.1.187
pc1012 B1 u28 port17 Cableid#3947 IP 10.65.1.188
pc1013 C5 u26 port24 Cableid#3410 IP 10.65.1.189
pc1014 D6 u36 port35 Cableid#23000064 IP 10.65.1.190

There are some issues with this task:

1. The hosts have been provisioned on Netbox with public IPs, see https://netbox.wikimedia.org/ipam/ip-addresses/?q=pc101
   
2. The sre.dns.netbox cookbook has not been run after the provisioning, this in turn has created the following issues:
   • The Icinga alert Uncommitted DNS changes in Netbox has fired and went unnoticed
   • Blocking future runs of the cookbook because the people running it will encounter an unwanted diff and might halt before proceeding being hence blocked or will go ahead and merge the changes creating wrong DNS records
3. Homer was not run for the affected switches, that in turn creates the following issues:
   • An email alert to rancid-core@ titled [Homer] Device live config differs from committed one
   • Pending changes on the switches that might block other people that will run homer finding the spurious diff they are not aware of.

I'm reverting the primary IP provisioning of those hosts and leaving only the already setup mgmt IPs and I'm running the cookbook to create the mgmt DNS records so to unblock the situation for everyone else.

Please be extra careful with the provisioning and do not let changes in Netbox sit without running the dns cookbook and homer right after. Also please keep an eye for the Uncommitted DNS changes in Netbox Icinga alert.
As an improvement we could make a cookbook that does all this in one single step.

Mentioned in SAL (#wikimedia-operations) [2021-07-12T12:42:20Z] <volans> reverting Primary IP allocation for pc1011-1014, leaving only mgmt IPs - T282484

I've updated the previous message with the homer part too. Given that we have the pending spurious changes that will open the ports for those hosts with the wrong vlan I'm reverting that part too in Netbox.

Those are the config I'm reverting if they are needed when it will be re-provisioned:

P.S. the cable ID 23000064 looks like a potential typo.

In T282484#7205448, @Volans wrote:

P.S. the cable ID 23000064 looks like a potential typo.

Not a typo, we just started ordering pre-serial-labeled network cables. Since its done at the factory, they are way longer numbers than what we printed ourselves.

Change 704854 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] pc101[1-4] mac addresses

https://gerrit.wikimedia.org/r/704854

Change 704854 merged by RobH:

[operations/puppet@production] pc101[1-4] mac addresses

https://gerrit.wikimedia.org/r/704854

In T282484#7215815, @RobH wrote:

In T282484#7205448, @Volans wrote:

P.S. the cable ID 23000064 looks like a potential typo.

Not a typo, we just started ordering pre-serial-labeled network cables. Since its done at the factory, they are way longer numbers than what we printed ourselves.

Ack, good to know, thanks. Please let me know in case those new numbers trigger any false positive in any Netbox report or other data validation that we have.

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['pc1011.eqiad.wmnet', 'pc1012.eqiad.wmnet', 'pc1013.eqiad.wmnet', 'pc1014.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107151934_robh_10629.log.

Completed auto-reimage of hosts:

['pc1014.eqiad.wmnet']

Of which those FAILED:

['pc1014.eqiad.wmnet']

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

pc1014.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202107152043_robh_32404_pc1014_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['pc1014.eqiad.wmnet']

Of which those FAILED:

['pc1014.eqiad.wmnet']

@RobH Sorry for delay cable was not seated completely in switch. fixed has link now

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

pc1014.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202107221235_kormat_1476_pc1014_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['pc1014.eqiad.wmnet']

and were ALL successful.

@RobH : i took the liberty to reimage pc1014 as the network connection is now working. I've also set it to 'staged' in netbox, but i'll leave this task for you to resolve in case there's anything more that has to be done. Cheers.