I've upgraded:

• the iDRAC version
• the NIC firmware
• the BIOS

I tried two versions of the NIC firmware, in case it was that, but nothing has made any difference.
When attempting to PXE boot, it never even tries to send a DHCP request, as far as I can see so far. I would expect to see DHCP traffic here.

@cmooney reported that:

the switch shows as no link / hard down, Linux seems to think it's working

I haven't checked this since the upgraded firmware, but the behaviour seems not to have changed at all.

The iDRAC also reports the port being up, but it hasn't received any packets.

I'm wondering if perhaps the SFP optical modules might not be working correctly, so it's not getting a two-way link.
That's just speculation though.

@BTullis we had the same issue with sessionstore2001 in codw see task below what we did was to replace the 1G RJ45/SFP converter i think it is the same issue since this server is using a 1G nic plug to a 10G switch. so i will advice we swap the SFP with another one and see if the switch link comes up. Same also with analytics1075 using 1G nic to connect to a 10G switch
https://phabricator.wikimedia.org/T340055

In T342141#9026115, @Papaul wrote:

@BTullis we had the same issue with sessionstore2001 in codw see task below what we did was to replace the 1G RJ45/SFP converter i think it is the same issue since this server is using a 1G nic plug to a 10G switch. so i will advice we swap the SFP with another one and see if the switch link comes up. Same also with analytics1075 using 1G nic to connect to a 10G switch
https://phabricator.wikimedia.org/T340055

@Papaul that would be great, please. As a possible alternative, I notice that these two servers do have unused dual-port 10 Gbps cards installed.
What about if we change which port is connected to the switch? We would benefit from an increase in throughput at the same time.
Do you think that's realistic to achieve?

Netbox has all ports listed as 1 Gbps, but I don't think that's correct.

I don't know how many other Hadoop workers are in the situation of having a 1 Gbps card connected to a 10 Gbps switch port, along with an unused 10 Gbps NIC installed. I'll see if I can find out.

@BTullis yes that is a possibility too to use the 10G nic since those 2 nodes each has 4x1G nic and 2x10G nic. There are 2 ways you will be able to switch to using the 10G nic on those servers.
1- Decommission the server and provision it again. this means the IP address will change on the server
2- manually change the IP configuration on the server. Remove the IP from the 1G nic and set it to the 10Gnic. if you want to keep the same IP address and update Netbox .

Another option if you have to keep using the 1G nic is to move the server within the same row in a 1G rack
let me know if you have any questions.

@Jclark-ctr - many thanks for doing that. I just checked with another run of the cookbook on analytics1073 and it doesn't look like the replacement sfpt has made any difference.

It still booted into its old O/S and it can't ping the default gateway.

root@analytics1073:~# uptime
 13:32:12 up 12 min,  1 user,  load average: 0.10, 0.11, 0.09
root@analytics1073:~# ping -c 3 10.64.21.1
PING 10.64.21.1 (10.64.21.1) 56(84) bytes of data.
From 10.64.21.117 icmp_seq=1 Destination Host Unreachable
From 10.64.21.117 icmp_seq=2 Destination Host Unreachable

--- 10.64.21.1 ping statistics ---
3 packets transmitted, 0 received, +2 errors, 100% packet loss, time 51ms
pipe 3

ethtool reports that the link is up though.

root@analytics1073:~# ethtool eno1
Settings for eno1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Full 
	                                     100baseT/Full 
	                                     1000baseT/Full 
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: off
	Supports Wake-on: g
	Wake-on: d
	Current message level: 0x000000ff (255)
			       drv probe link timer ifdown ifup rx_err tx_err
	Link detected: yes

There are 2 ways you will be able to switch to using the 10G nic on those servers.
1- Decommission the server and provision it again. this means the IP address will change on the server
2- manually change the IP configuration on the server. Remove the IP from the 1G nic and set it to the 10Gnic. if you want to keep the same IP address and update Netbox .

Thanks @Papaul

If it's not too much trouble, I'd like to try going for option 2 on these two servers, please. Is that OK with you too @Jclark-ctr ?

I'll update /etc/network/interfaces on analytics1073 and analytics1075, changing the network device name from eno1 to enp130s0f0
Should I update the interfaces on the servers in netbox, or are you better placed to do that?

the interface came up an went down

papaul@asw2-b-eqiad> show interfaces descriptions ge-7/0/15
Interface       Admin Link Description
ge-7/0/15       up    up   analytics1073

papaul@asw2-b-eqiad> show interfaces descriptions ge-7/0/15
Interface       Admin Link Description
ge-7/0/15       up    down analytics1073

right now 1075 is showing up

papaul@asw2-c-eqiad> show interfaces descriptions | match analytics1075
ge-7/0/5        up    up   analytics1075

In T342141#9028025, @Papaul wrote:

right now 1075 is showing up

papaul@asw2-c-eqiad> show interfaces descriptions | match analytics1075
ge-7/0/5        up    up   analytics1075

OK, what should I do to help troubleshoot? Should I kick off a reimage cookbook for analytics1075 or a sre.hosts.dhcp cookbook?
I haven't upgraded any of the firmware on analytics1075 yet, but I can do that if you like.

We've been investigating this extensively and discussing in some depth on #wikimedia-dcops on IRC.

We've decided to go ahead and upgrade analytics1073 and analytics1075 from their 1 Gbps nics to their unused 10 Gbps nics.

With luck, @Jclark-ctr will be able to change the Cat5 cable for a DAC cable tomorrow. @cmooney has made the requisite changes in netbox and has already updated /etc/network/interfaces on the hosts with the new interface names.
These names are likely to change under bullseye, which makes this a bit of a tricky process. We're going to try to boot into buster and verify that we have a link reported consistently on both ends, then attempt a reimage.

@BTullis replaced cable on analytics1073 & analytics1075