Called HP at 17:48 CDT call ended at 18:25 CDT. went over again the new Log from today and didn't see any internal problem on the server. The only thing he asked me to do it to change the power settings of the server from Dynamic power saving mode to Static High performance mode. The error reported on the integrated management log according (Critical Temperature Threshold Exceeded) to him is an external issue.

The server reset again. This time I have something in the log file. see below

@fgiunchedi HW diagnostics didn't report any problem. Upgrade all firmware on the server . we can observe the server all this week and if we do have the same problem again, I will contact HP.

@fgiunchedi getting "The last firmware update attempt was not successful. Ready for the next update." error when trying to update the BIOS. so I email the HP engineer and waiting for response.

Engineer called at 2:20 PM CDT called ended at 2:37 PM CDT. He went over the AHS logs and didn't find any error or issue. His recommendation as I mentioned was to upgrade the BIOS and the controller. @fgiunchedi let me know.

@fgiunchedi I open a case with HP and for now They said there is no engineer available to help me, I will be receiving a call back in an hour. Please see below for case information.

@fgiunchedi nerver seen this on HP servers. I also checked the log it just says "server reset" no other reason. Since the server is under warranty, I can open a case with HP and submit them the AHS logs they can provide me with more information after they review the log.
If they don't find anything, they will ask me o upgrade the server firmware.

@Jgreen it is all yours. Ping me when you are ready to do the install so you can show me how you do it. Will like to help with install in the future. Thanks.

I talked with @Jgreen on IRC, he said to use the IP address of frmon2001 for frauth2001 for now since frmon2001 is not installed yet.

Done

@Jgreen @Dzahn has a comment on https://gerrit.wikimedia.org/r/#/c/operations/dns/+/460127/ the network 10.195.0.72/29 can only hold 6 hosts (2 at the power 3 ) - 2 = 6 . the first usable IP address for that network is 10.195.0.73 and the last usable address is 10.195.0.78 so we can not use 10.195.0.79

• Re-seat memory
• Upgrade BIOS from version 2.3 to 2.6
• Upgrade IDRAC from version 1.4 to 2.60

Server is back up

@fgiunchedi this server is out of warranty. The IDRAC log also is not showing any memory errors and the firmware on the server is way out of date. The first options will be to upgrade the firmware on the server and reboot the severs 2 or 2 times to see .The second option if we stay have the same problem is to pull a memory stick from one of the decommissioned servers.

Disk replaced

@MoritzMuehlenhoff i will leave it to you so you can play with it tomorrow

Here what I get now

@MoritzMuehlenhoff ok I will change the install in DHCP

@MoritzMuehlenhoff here is what i get

@ayounsi I am planning on going to eqdfw this Thursday at 10:30am CDT

sure

@Dzahn you can disable those alerts @MoritzMuehlenhoff is running some test on that server.

@MoritzMuehlenhoff both the 10GB and 1GB NIC's are already connected to the switch

First puppet run complete

@MoritzMuehlenhoff the installation is complete it is all yours

@MoritzMuehlenhoff I changed the switch to use the ge-2/0/12 instance of xe-2/0/12 since we are using a 1GB transceiver. the installation is in progress i will let you know when installation is done.

show ethernet-switching table interface ge-2/0/12

@Gehel Disks added to wdqs2001

Disks are in place

BIOS upgrade from version 2.4.3 to 2.8.0
IDRAC upgrade from version 2.40 to 2.60

Disk replacement complete

Done

complete

@Jgreen Server is back up please confirm and close task.
Thanks.

@ayounsi Please see below

Dear Juniper Networks Customer,

@ayounsi All lights on all PSU's on cr2 are green. What do you want me to do here?

This is what was showing

• Drain the power
• Upgrade BIOS from version 2.3.3 to 2.6.0
• Upgrade IDRAC from 1.4.2 to 2.60

server is back up

Complete

Disk wipe in progress

on boad NICs disable

Disk replaced

switch port information
ge-3/0/16

show interfaces ge-3/0/16 
Physical interface: ge-3/0/16, Administratively down, Physical link is Down
  Interface index: 1187, SNMP ifIndex: 557
  Description: DISABLED

This is complete resolving it.

Disk wipe in progress

@ayounsi all SFP+-10G-LR are in place .

@Marostegui complete closing the task. Thanks

switch port is enable and link is up

show interfaces ge-1/0/17 descriptions 
Interface       Admin Link Description
ge-1/0/17       up    up   labtestnet2003:eth1

papaul@asw-b-codfw# show interfaces interface-range cloud-instance-ports 
member ge-5/0/32;
member ge-5/0/31;
member ge-5/0/30;
member ge-1/0/16;
member ge-1/0/17;
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members cloud-instances1-b-codfw;
        }
    }
}

switch port information

Moved all disks in one of the decom server (db2013). Disk wipe in progress

yes thanks

Done

@Marostegui Bios update to 2.8.0 . It is all yours

This server has pro support as mentioned in T139283

@Marostegui Disk replaced

@Andrew both ports are up and in the cloud-instance-ports interfaces ranges. Please check if everything looks good, you can resolve the task.

Port information
labtestnet2002 rack B1 ge-1/0/16
labtestnet2003 rack B1 ge1/0/17