I have recently been doing some work around IPMI SEL and have noticed that theses hosts:
Are spamming the SEL logs with messages like the following
@wiki_willy all 3 of these servers are well out of warranty (2-3 years). analytics1068 is marked failed in netbox
@Jclark-ctr has a few spares onsite, so we can probably use those as replacements. Thanks, Willy
@jbond i have batteries for all of these can this be done tomorrow? If possible can you shut down server and I can preform repair 9am est tomorrow?
@Jclark-ctr you will need to contacts someone in analytics (possibly @BTullis) and data persistence (maybe @MatthewVernon)
@Jclark-ctr I can shut down ms-be1042 for you (or you can DIY, there's no special procedure for this host). Can I confirm you want it shut down at 14:00 UTC today, please?
Icinga downtime and Alertmanager silence (ID=f76e48e4-3716-4c3a-8992-2858603cabe9) set by btullis@cumin1001 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Replacing battery
an-worker1091.eqiad.wmnet
Mentioned in SAL (#wikimedia-analytics) [2023-03-31T08:44:06Z] <btullis> Shutting down an-worker1091 for RAID battery replacement T332883
@Jclark-ctr I've shut down an-worker1091 so you can replace the battery at any time. Feel free to boot it when the work is finished, as it should rejoin the cluster with no additional intervention needed.
analytics1068 is scheduled for decommissioning so I wouldn't waste a fresh battery on it at this point. We can add extended downtime for the MegaRAID icinga check, until we get around to the decom.
Many thanks.
Icinga downtime and Alertmanager silence (ID=e19efa89-db0e-4ad2-bcc9-ed867218f629) set by mvernon@cumin2002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Add-in Card 2 ROMB Battery LOW
ms-be1042.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2023-03-31T11:42:17Z] <Emperor> shutdown ms-be1042 for battery swap T332883