I randomly encountered this while manually checking maps2009. Just a heads up this node is a master node which means that if services are down, OSM data syncing/invalidation and postgres replication is down and the cluster will be out of sync.

@Joe @Jgiannelos maps2009 doesn't look to be up at all. the network link and mgmt link are down and the front panel indicates that the server is off. anyway you can confirm?

after attempting to boot the server I believe it is a bad motherboard. server will not power up even with minimum configuration. PSUs/PDU are working/have green lights. No network lights, no idrac indicators, no front panel activity etc. No Console activity.

I have created a dispatch with Dell. The server is out of warranty on the 27th of this month. So good thing it went out now and not in September.

Service Request 173930280

Dell sent me a list of checks to determine if it's the motherboard or the backplane. followed directions and replied. my guess is the MB will need to be replaced. will update when part arrives.

@Jhancock.wm thank you very much for the update!

Mentioned in SAL (#wikimedia-operations) [2023-08-16T10:45:19Z] <effie> depooling maps on codfw - T344110

replaced the system board and the controller. System still did not post. pulled out everything except 1 ram, 1 cpu, a psu. Booted and started adding back components. Found that the backplane is causing the system to not boot. have contacted Dell and let them know. expecting another part soon. will update as events unfold.

@Jhancock.wm thanks for working on this. since we have some R440 in storage can you pull the backplane out of one of those servers and try while we waiting for Dell to send us one?

Thanks.

I can give it a shot. Dell confirmed they will be sending a new part this morning.

@Jhancock.wm in that case just wait for the new part. thanks

The backplane has been replaced and the settings in idrac have been updated. I think it's good to go back. I will leave this ticket up for a day to observe. Please let me know if you come across any issues that I need to check!

On the OSM sync side of things, it might worth checking if the system catches up with the diffs (~1 week worth of diffs could be manageable). The idea is that we avoid having a full planet import from scratch for codfw.

Turns out there is one more thing I need to do to. I missed a firmware update. Is it safe for me to reboot at this time?

In T344110#9117302, @Jhancock.wm wrote:

Turns out there is one more thing I need to do to. I missed a firmware update. Is it safe for me to reboot at this time?

@Jhancock.wm, I dropped the ball on this one, I will ping you and give you a window when it will be safe to reboot. Thank you very much for your effort on this!

@jijiki all good. I knew it needed time to do its thing. I'll be on site for the next 3 hours today and roughly the same time for the rest of the week. thanks!

@jijiki is there a day this week that I can update the firmware on this server?

In T344110#9143064, @Jhancock.wm wrote:

@jijiki is there a day this week that I can update the firmware on this server?

Hey @Jhancock.wm we need a slight coordination which has not been possible to do so yet, sorry for this taking longer, I will update you as soon as we can

@jijiki I haven't seen the idrac/ssh go down on this server in a while. is it ok if I close this ticket? if there's still work that needs to be done I can hold onto it. =)