CyrusOne Checked the reading from the fiber patch panel in A8 same readings. So they are still going to run some test out of the cage.

@fgiunchedi is it possible to depool this server for me to do a firmware upgrade before I resolve the task?

disk replaced

The network cable was was plugged back in after the disk replacement. Should be good now.

db2089 upgrade complete
Upgrade
BIOS from 2.4.3 to 2.9.1
IDRAC from 2.40. to 2.61

Can db2089 be depool please if it is not yet? Thanks

Upgrade
BIOS from 2.4.3 to 2.9.1
IDRAC from 2.40. to 2.60

@Marostegui this can be done anytime today. Just let me know when the server is down. Thanks

@Marostegui in most cases the CPU1/CPU2 Machine check error detected is caused from outdated BIOS. I will recommend that we first update the BIOS. The system BIOS right now is at 2.4.3 and there is a new version out (2.9.1) from 11/02/2019.After this we can check some settings in the BIOS under BIOS profile .

Complete

Previous hardware has been already returned since last Thursday. (See comment on Feb7) We can resolve this task.

Checked temperature in the rack all looks good. add blanks to the rack since we have only 8 servers in that rack. Leaving the task open for another week.

complete

Old server has been shipped out. Shipping information below.

@aborrero @Andrew all yours . Let me know if you have any questions.

second NIC configuration

Put back sda in the server.

• Remove sda from the server
• boot the server
• server boot without a problem

Disk replaced, server didn't boot up.

@fgiunchedi I replaced the problematic server with the new one Dell shipped to me. The OS is installed and puppet first run done. I will proceed to the disk wipe on the old server on Wednesday before shipping it back to Dell. Let me know if you have any questions.

I put back the bad disk and boot the system and the system boot into OS with no problem. it looks like what @jcrespo and other mentioned on IRC the grub is installed on /dev/sda/ only which is the disk that needs to be replaced. so we need to fix this issue first so I can be able to replace the disk.

Disk with serial number WMAYP0E607DT has been replaced. Server can not find boot device. Server can not boot to OS after disk replacement.

Can you please update this disk with which disk failed? Thanks

Removed old puppet cert for ms-be2047.codfw.wmnet

update Netbox with new serial number

Received replacement server

@Andrew there is no raid controller on the new servers. They all have 2x200GB SSD's

@Andrew can you also specify on this task in which VLAN eth1 needs to be for cloudvirt200[1-3]. Thanks

@Andrew for all those new servers I am using for partman labvirt-ssd.cfg?

@Marostegui disk replacement complete

looks like the mgmt switch froze have to unplug and plug the power back. Switch is back up

This is complete. All servers ready to be ship out.

This is complete. All servers ready to be ship out.

papaul@asw-c-codfw> show chassis environment | match Power
Power FPC 1 Power Supply 0 OK

FPC 1 Power Supply 1           OK        
FPC 2 Power Supply 0           OK

BIOS from 2.4.3 to 2.8.0
IDRAC from 2.40 to 2.61

Disk replacement complete

Dell just called me. They will be shipping a new system and will arrive by the first week on January.

Power cable got loose as well may be when working on asw-b8-codfw on Tuesday. System is back up.

Loose power cable. System is back up.

Redundancy Policy on this system was set to Not redundant or on the other working system it was set to redundant so we change the settings for this system to redundant as well. Monitoring the system again

Firmware upgrade complete

fpc2-fpc8 xe-2/0/41 and xe-2/0/42
fpc7-fpc8 xe-7/0/43 and xe-7/0/44

Before

papaul@asw-c-codfw> show interfaces descriptions | match "ge-1/0/1[0-2]"     
ge-1/0/10       up    down elastic2013
ge-1/0/11       up    down elastic2014
ge-1/0/12       up    down elastic2015

The problem happen again twice after replacing CPU1

connected to scs-c1-codfw on port 48

Complete

fpc2-fpc8 connection xe-2/0/41 and xe-2/0/42
fpc7-fpc8 connection xe-7/0/43 and xe-7/0/44

CPU 1 has been replaced. I clear also the log. The system is back up and I will be monitoring it once again.

Dell will be shipping 1 New CPU by Monday.

This can be resolved then since i am done with it .

This is complete

same error again at 22:47

@Marostegui no need to close the task. It can be assign to @RobH so he can keep track

@Marostegui any reason why production DNS is still showing for pc2004?