Page MenuHomePhabricator
Create Task
Maniphest T347287

Replace RAID controller battery on an-worker1086
Closed, ResolvedPublic
Actions

Description

We have experienced another RAID controller battery failure from a batch of hosts that have all been failing at similar times.

This time is is an-worker1086

Please would you replace the RAID controller battery, when convenient?

The server is still functioning, but with reduced performance as the cache is operating in a WriteThrough mode

image.png (316×720 px, 50 KB)

https://alerts.wikimedia.org/?q=%40state%3Dactive&q=team%3Dsre&q=instance%3Dan-worker1086

It can be shut down at any time, without any special procedure required. Similarly, once the battery has been replaced it will rejoin the Hadoop cluster on boot, without any special action.

Related Objects
Search...

Event Timeline

BTullis triaged this task as Medium priority.Sep 25 2023, 12:22 PM
BTullis created this task.
BTullis added a project: Data-Platform-SRE.
BTullis moved this task from Incoming to Blocked / Waiting on the Data-Platform-SRE board.
Maintenance_bot added a project: SRE.Sep 25 2023, 12:29 PM
Jclark-ctr subscribed.Sep 25 2023, 2:20 PM

@BTullis i am on site today. otherwise we can do it tomorrow

ops-monitoring-bot added a comment.Sep 26 2023, 10:37 AM

Icinga downtime and Alertmanager silence (ID=d9fc4cc1-c0d4-4a6d-83d0-127f1d08a401) set by btullis@cumin1001 for 3 days, 0:00:00 on 1 host(s) and their services with reason: Downtiming host for RAID controller battery replacement

an-worker1086.eqiad.wmnet
BTullis added a comment.Sep 26 2023, 10:38 AM

I have shut down the host, so it is ready for work. Feel free to boot it normally when finished. Thanks @Jclark-ctr.

Jclark-ctr claimed this task.Sep 26 2023, 2:11 PM

@BTullis Replaced Raid controller battery server is coming back up now

Jclark-ctr added a comment.Sep 26 2023, 2:14 PM

This is second battery replacement for this server. T326127 was 1st. although battery did not physically look bad i did still replace it. If it returns we might need to look at other issues with possibly raid card

Jclark-ctr closed this task as Resolved.Sep 26 2023, 7:41 PM
BTullis added a comment.EditedSep 27 2023, 12:26 PM

Just adding here, the server didn't boot successfully.
It firstly said that there was a foreign RAID config for one of the drives.
After this was imported it complained of disk errors.
{F37816784,width=60%}

Maybe I'll reopen this ticket to look at the disk issue.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL