@Dwisehaupt two network connections have now been provisioned. lmk if you need anything else =)

the four servers in codfw have had cables physically removed and deleted in netbox.

all ports verified empty and removed from netbox

@jcrespo these two servers have been had their bios upgraded to 2.7.5. please let us know if you have any other issues with these!

i fixed the one in d5, but gonna physically inspect the ones in c8 cause of an overabundance of caution.

already had a separate task but thank you!

@Dwisehaupt did you get my email about the password?

np!

@Jgreen hey lost track of this task. I'm ususually on site from 9am to 1pm local time (central) on almost every work day. is there a time that would work best for y'all?

@MoritzMuehlenhoff thanks for the help and correction. 22353BB15C0C has been replaced.

@MoritzMuehlenhoff the replacement has arrived. can you confirm that its safe to replace the drive at this time.
Also can you help confirm that it's second drive that needs to be replaced? I'm 80% sure but would appreciate the nod.

@MatthewVernon drive has been replaced.

@MatthewVernon drive has arrived. please let me know if it's okay to replace the drive at this time.

got the replacement rolling with dell. SR219265258

@MatthewVernon drive is for sure faulty. even shows up in idrac. started process to get it replaced by dell since the server is in warranty. SR219219399.

logged in to idrac to check. so far so good. if it doesn't alert by monday, we should be able to close the ticket.

@Marostegui i rotated DIMM_A6 with DIMM_A10 to see if the error follows the stick. unfortunately, we do have to wait for it to happen again to diagnose it. Since the cpu errors are floating but the DIMM errors stay with A6 so far, I'm willing to bet that it's the stick.

@Andrew could you or someone on your team fill out the Hostname and racking details for us? And also make any needed updates to the site.pp and preseed files? Thank you for your help!

@MatthewVernon thanks for your help!

@MatthewVernon got the main issue with this one fixed. it failed cause it's trying to reach the wrong puppet server. Gonna take another crack this afternoon but feel free to take it away from me if you sign on before i finish it.

@MatthewVernon avoiding reopening the ticket for metric reasons. did this one need any additional attention? i was holding onto the bad drive until you got back.

Is this a false alert? I'm not seeing any issues physically with the server or in the idrac.

balanced power

@bking do you need me to set up anything in codfw at this time?

@Raine ran into a secondary issue with the backplane, but it's fixed now. let us you know if you run into any other issues.

note to self: 90 needs the mgmt connection checked for connectivity.

server is decommed.

@MatthewVernon drive has been replaced.

power balanced.

they shipped the drive today after escalating! i'll plug this in first thing when it gets here. should be here thursday.

@MatthewVernon ms-be208[5-7] have had the controllers replaced and i waited long enough to make sure the drives appeared after booting up.

fyi Dell is fighting me on this cause the idrac doesn't show a failure. so any extra evidence you got to throw at them would be appreciated. I've already asked for an escalation but i expect it to come back again.

balanced power

@Andrew please fill out the racking details in this task and make any updates that are needed for the preseed and site files. Thank you!

process started with dell: SR218123931

@MatthewVernon
I have an 8tb replacement drive, but the sata speed is only 6 Gbps instead of 12. will this work? if not i can get a replacement from Dell pretty quickly. Gonna start that process anyway since the server is under warranty and we'll want a replacement on hand.

balanced power

finally got dell to ship me the parts. should have time to take care of the replacement monday or tuesday next week.

missed this one getting the new alert limits set. fixed.

i ran reset /system1/pwrmgtsvc1 with a physical console up to observe. it didn't reboot for me.