Page MenuHomePhabricator
Create Task
Maniphest T258910

Remove db1082's BBU on-site
Closed, ResolvedPublic
Actions

Description

We've been having this host crashing twice due to BBU issues (T258336)
The BBU still doesn't show as disabled or gone by the OS or the ILO, and I haven't found a way to do it.
I believe that we'll still see more crashes if we don't remove/disable it.

Other hosts that had broken BBU, as soon as it is "gone" and not show anymore, they are "fine", so I would like to force this host to do so.

HW logs for both crashes are:

description=Smart Storage Battery has exceeded the maximum amount of devices supported (Battery 1, service information: 0x07). Action: 1. Remove additional devices. 2. Consult server troubleshooting guide. 3. Gather AHS log and contact Support

Can we manually remove/disable it on site?
This host is scheduled to be decommissioned in Q2, but I am trying to work out if we can accelerate the purchase of its replacement in Q1 instead.

Related Objects
Search...

Event Timeline

Marostegui created this task.Jul 27 2020, 5:43 AM
Marostegui triaged this task as High priority.
Jclark-ctr added a comment.Jul 27 2020, 4:03 PM

@Marostegui is host down now? i can remove in about 1 hour

Stashbot added a comment.Jul 27 2020, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-27T16:04:31Z] <marostegui> Stop MySQL on db1082 for onsite maintenance - T258910

wiki_willy assigned this task to Jclark-ctr.Jul 27 2020, 4:05 PM
Marostegui added a comment.Jul 27 2020, 4:06 PM

@Jclark-ctr the host is now off, you can proceed whenever you want
Thank you!

Jclark-ctr added a comment.Jul 27 2020, 5:04 PM

@Marostegui removed bbu. host is booting up now

RhinosF1 mentioned this in T258965: Degraded RAID on db1082.Jul 27 2020, 5:17 PM
Marostegui closed this task as Resolved.Jul 27 2020, 6:28 PM

Thanks - I can see it:

root@db1082:~# hpssacli controller all show detail | grep -i battery
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0

Thank you guys for the fast response!

Marostegui mentioned this in T258336: db1082 crashed.Jul 28 2020, 10:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits