Page MenuHomePhabricator
Create Task
Maniphest T227408

(OoW) restbase2009 lockup
Closed, ResolvedPublic
Actions

Description

restbase2009 has been down since 2019-07-07 8:40, on power off / on a message appeared on console about battery failure from the controller (to be investigated)

Event Timeline

fgiunchedi created this task.Jul 7 2019, 3:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2019, 3:21 PM
Reedy added a project: ops-codfw.Jul 7 2019, 4:47 PM
jijiki triaged this task as Medium priority.Jul 8 2019, 5:32 AM
jijiki added a project: serviceops.
jijiki added a subscriber: jijiki.
jijiki added subscribers: Eevans, • Pchelolo.Jul 8 2019, 5:35 AM
wiki_willy renamed this task from restbase2009 lockup to (OoW) restbase2009 lockup.Jul 15 2019, 7:39 PM
wiki_willy assigned this task to Papaul.
Papaul added a comment.Jul 17 2019, 4:08 PM

Indeed the server is not showing the Smart Storage Battery status. Lets try to upgrade the server firmware since the last upgrade was from 2015.

@fgiunchedi Let me know when we can depool this server for firmware upgrade.

Thanks

Papaul added a comment.Jul 17 2019, 4:11 PM

Screenshot from 2019-07-17 11-10-00.png (121×1 px, 13 KB)

Eevans added a comment.Jul 17 2019, 4:58 PM

@Papaul you can take the server down as needed.

Papaul added a comment.Jul 17 2019, 6:14 PM

After Firmware upgrade, we still have the Smart storage battery problem since the server is out of warranty we can not have the part replaced.

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Jul 17 2019, 8:27 PM
jijiki moved this task from Incoming 🐫 to Unused 🥌 on the serviceops board.Jul 19 2019, 8:56 AM
jijiki added a comment.Jul 19 2019, 9:59 AM

@Eevans Shall we mark restbase2009 as inactive on conftool?

Eevans added a comment.Jul 24 2019, 7:01 PM
In T227408#5349035, @jijiki wrote:

@Eevans Shall we mark restbase2009 as inactive on conftool?

I'm not positive I understand the implications of that.

As far as I know, the host went down, and was rebooted through the management console. Later it was taken down again for a firmware update to address the smart storage battery fault (which did not clear the fault). As of this moment, the machine is up and running. What would marking it inactive do?

jijiki added a comment.Jul 25 2019, 9:24 AM

@Eevans I was under the impression we have more work to be done on the server. Shall we mark this task as resolved?

Eevans added a subscriber: PPaul.Jul 26 2019, 1:55 AM
In T227408#5364314, @jijiki wrote:

@Eevans I was under the impression we have more work to be done on the server. Shall we mark this task as resolved?

I was under that impression too. @PPaul's last comment indicated that there is a problem with the Smart storage battery, but that the machine is out of warranty. What do we do in such a situation?

jijiki added a comment.EditedJul 26 2019, 12:34 PM

@Papaul Can you let us know what are our options (if any?)

PPaul removed a subscriber: PPaul.Jul 26 2019, 3:07 PM
Papaul reassigned this task from Papaul to wiki_willy.Aug 1 2019, 1:32 PM
Papaul added subscribers: wiki_willy, Papaul.

@jijiki I will talking to @wiki_willy to see what are our options on this.

@wiki_willy this system is out if warranty since April 2019 and we do have a problem with the Smart storage battery. The option I have here is, We do have 5 HP servers that were decom I can look and see if those servers have the same Smart storage battery with this system. In case this is not the case, can you please advice.

Thank you.

wiki_willy added a comment.Aug 1 2019, 4:04 PM

@Papaul - if you can't find a spare from any of those decom servers, we can order it, since it's still a while before the 5yr mark.

Thanks
Willy

wiki_willy reassigned this task from wiki_willy to Papaul.Aug 2 2019, 1:04 AM
Stashbot added a comment.Aug 5 2019, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-05T14:06:16Z] <jijiki> Depool and restart restbase2009 for maint - T227408

Papaul added a comment.Aug 5 2019, 2:10 PM

@jijiki i need this serveur power down
thanks

jijiki added a comment.Aug 5 2019, 2:10 PM

@Papaul Server is depooled, ping me when do pool it back, many thanks !

Papaul added a comment.Aug 5 2019, 3:55 PM

@jijiki please repool the server when you have a minute. We will have to order a new Storage battery for the server since all the decom HP servers are GEN8 and this one is a GEN9 so different storage battery.

Thanks.

Papaul mentioned this in Unknown Object (Task).Aug 5 2019, 4:09 PM

@jijiki I made a procurement task the the storage battery at T229847

Stashbot added a comment.Aug 5 2019, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-05T17:33:42Z] <jijiki> Pool restbase2009 - T227408

jijiki closed this task as Resolved.Aug 5 2019, 5:34 PM

@Papaul I am marking this as resolved, thank you!

BBlack reopened this task as Open.Sep 11 2019, 5:02 PM
BBlack added a subscriber: BBlack.

Re-open as this isn't really complete yet, the battery came in and replacement is proceeding. Since @jijiki did this before and claims it's just a depool command, we'll go with that again :)

Stashbot added a comment.Sep 11 2019, 5:05 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T17:05:11Z] <bblack> restbase2009 - depool for hardware work - T227408

Stashbot added a comment.Sep 11 2019, 5:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T17:07:20Z] <bblack> restbase2009 - shutdown for hardware work - T227408

Papaul added a comment.Sep 11 2019, 5:19 PM

Smart storage replacement complete.
Embedded HPE Smart Storage Battery 875241-B21 878643-001 6WQXL0BB2BQ4H8 01 0.60 OK

Stashbot added a comment.Sep 11 2019, 5:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T17:27:32Z] <bblack> restbase2009 - re-pool - T227408

Papaul closed this task as Resolved.Sep 11 2019, 5:59 PM

This can be resolved

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL