db1094 crash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Mar 18 2017, 6:22 PM

Description

This led to a full system crash- it did not even let the os continue after the disk failed:

</>hpiLO-> show system1/log1/record13

status=0
status_tag=COMMAND COMPLETED
Sat Mar 18 19:48:36 2017



/system1/log1/record13
  Targets
  Properties
    number=13
    severity=Caution
    date=03/18/2017
    time=17:53
    description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support
  Verbs
    cd version exit show


</>hpiLO-> show system1/log1/record14

status=0
status_tag=COMMAND COMPLETED
Sat Mar 18 19:48:45 2017



/system1/log1/record14
  Targets
  Properties
    number=14
    severity=Caution
    date=03/18/2017
    time=18:22
    description=POST Error: 313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists.
  Verbs
    cd version exit show

On reboot:

313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400.
Action: Restart system. Contact HPE support if condition persists.

Details

Subject	Repo	Branch	Lines +/-
mariadb: Pool db1094 with full weight	operations/mediawiki-config	master	+1 -1
mariadb: Increase db1094 weight after initial pool	operations/mediawiki-config	master	+1 -1
mariadb: Repool db1094 after crash	operations/mediawiki-config	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T160833: Reduce verbosity of DBReplication logs from non-debug requests

Event Timeline

jcrespo created this task.Mar 18 2017, 6:22 PM

Restricted Application added a project: SRE. · View Herald TranscriptMar 18 2017, 6:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 343451 had a related patch set uploaded (by Jcrespo):
[operations/mediawiki-config] mariadb: Depool db1094 after crash

https://gerrit.wikimedia.org/r/343451

gerritbot added a project: Patch-For-Review.Mar 18 2017, 6:29 PM

jcrespo mentioned this in T160833: Reduce verbosity of DBReplication logs from non-debug requests.Mar 18 2017, 6:47 PM

Did it page?? I never got any page or was it frozen but mysql was still up but unresponsive?

It didn't page. db1094 web interface is impossible to access- while I can access with no problem to db1093 and db1095. :-/

jcrespo removed a project: Patch-For-Review.Mar 18 2017, 7:50 PM

jcrespo updated the task description. (Show Details)

Apart from contacting HP we might as well upgrade the firmware if possible anyways:

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c05187655&sp4ts.oid=7252820

jcrespo updated the task description. (Show Details)Mar 20 2017, 9:44 AM

Change 343607 had a related patch set uploaded (by Jcrespo):
[operations/mediawiki-config] mariadb: Repool db1094 after crash

https://gerrit.wikimedia.org/r/343607

gerritbot added a project: Patch-For-Review.Mar 20 2017, 10:01 AM

Change 343607 merged by jenkins-bot:
[operations/mediawiki-config] mariadb: Repool db1094 after crash

https://gerrit.wikimedia.org/r/343607

Change 343937 had a related patch set uploaded (by Jcrespo):
[operations/mediawiki-config] mariadb: Increase db1094 weight after initial pool

https://gerrit.wikimedia.org/r/343937

Change 343937 merged by jenkins-bot:
[operations/mediawiki-config] mariadb: Increase db1094 weight after initial pool

https://gerrit.wikimedia.org/r/343937

Change 343955 had a related patch set uploaded (by Jcrespo):
[operations/mediawiki-config] mariadb: Pool db1094 with full weight

https://gerrit.wikimedia.org/r/343955

Change 343955 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Pool db1094 with full weight