Page MenuHomePhabricator
Create Task
Maniphest T205514

db1092 crashed - BBU broken
Closed, ResolvedPublic
Actions

Description

db1092 got frozen while a heavy alter got replicated from the master.

This is what we have on the HW logs:

</system1/log1>hpiLO-> show record14

status=0
status_tag=COMMAND COMPLETED
Wed Sep 26 07:44:43 2018



/system1/log1/record14
  Targets
  Properties
    number=14
    severity=Caution
    date=09/26/2018
    time=06:05
    description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support
  Verbs
    cd version exit show

Update:
@Cmjohnson: Jaime kindly downloaded the logs for Support so you have them at: T205514#4618246

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+1 -1db-eqiad.php: Increase weight for db1092
operations/mediawiki-configmaster+2 -2db-eqiad.php: Repool db1092 with low weight
operations/mediawiki-configmaster+1 -1db-eqiad.php: Clarify db1092 status
operations/mediawiki-configmaster+4 -3mariadb: depool db1104
operations/puppetproduction+1 -0mariadb: Disable notifications on db1092, crashed
operations/mediawiki-configmaster+5 -5db-eqiad.php: Depool db1092
Customize query in gerrit

Related Objects

Duplicates Merged Here
T205516: Degraded RAID on db1092

Event Timeline

Marostegui created this task.Sep 26 2018, 7:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2018, 7:50 AM
Stashbot added a subscriber: Stashbot.Sep 26 2018, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-26T07:50:51Z] <marostegui> Hard reset db1092, server crashed - T205514

Marostegui added a comment.Sep 26 2018, 7:53 AM

On reboot:

313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400.
Action: Restart system. Contact HPE support if condition persists.
gerritbot added a subscriber: gerritbot.Sep 26 2018, 7:59 AM

Change 462872 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/462872

gerritbot added a project: Patch-For-Review.Sep 26 2018, 7:59 AM
gerritbot added a comment.Sep 26 2018, 8:02 AM

Change 462872 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/462872

Stashbot added a comment.Sep 26 2018, 8:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-26T08:04:17Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1092, server crashed - T205514 (duration: 00m 56s)

gerritbot added a comment.Sep 26 2018, 8:04 AM

Change 462873 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable notifications on db1092, crashed

https://gerrit.wikimedia.org/r/462873

Marostegui assigned this task to Cmjohnson.Sep 26 2018, 8:06 AM
Marostegui triaged this task as Medium priority.
Marostegui added a project: ops-eqiad.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a subscriber: Cmjohnson.

@Cmjohnson looks like we need a new BBU. This host is under warranty, can you talk to HP and see if we can get a new BBU before 10th Oct (as that is the DC failover scheduled date).
Thanks!

Restricted Application added a project: SRE. · View Herald TranscriptSep 26 2018, 8:06 AM
gerritbot added a comment.Sep 26 2018, 8:15 AM

Change 462873 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable notifications on db1092, crashed

https://gerrit.wikimedia.org/r/462873

Marostegui renamed this task from db1092 crashed to db1092 crashed - BBU broken.Sep 26 2018, 8:19 AM
Marostegui merged a task: T205516: Degraded RAID on db1092.
Marostegui added a subscriber: ops-monitoring-bot.
jcrespo added a subscriber: jcrespo.EditedSep 26 2018, 8:31 AM

IML.csv2 KBDownload

AHS log on google drive, cannot attach it on phabricator: https://drive.google.com/file/d/1Y9RikXhRlZHY-7gNN0MeK5AdOjswPXJH/view?usp=sharing

Marostegui updated the task description. (Show Details)Sep 26 2018, 8:37 AM
Cmjohnson added a comment.Sep 26 2018, 2:31 PM

A support ticket has been submitted with HPE

Case ID: 5332806955

Marostegui added a comment.Sep 26 2018, 2:31 PM

Thanks!! :-)

Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Sep 26 2018, 2:32 PM
• Banyek added a project: User-Banyek.Sep 26 2018, 3:11 PM
• Banyek added a subscriber: • Banyek.

I will reclone this database instance

• Banyek moved this task from Backlog to next on the User-Banyek board.Sep 26 2018, 3:34 PM
• Banyek added a comment.Sep 27 2018, 8:30 AM

The donor host will be db1104 for recloning, and I update that first

Stashbot added a comment.Sep 27 2018, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T08:30:16Z] <banyek> upgrading db1104 (kernel-mariadb) and rebooting it (T205514)

Stashbot added a comment.Sep 27 2018, 8:56 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T08:56:41Z] <banyek> stopping replocication & mariadb on db1104 and db1092 as db1092 is getting recloned from db1104 (T205514)

gerritbot added a comment.Sep 27 2018, 9:23 AM

Change 463219 had a related patch set uploaded (by Banyek; owner: Banyek):
[operations/mediawiki-config@master] mariadb: depool db1104

https://gerrit.wikimedia.org/r/463219

gerritbot added a comment.Sep 27 2018, 9:51 AM

Change 463219 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: depool db1104

https://gerrit.wikimedia.org/r/463219

Stashbot added a comment.Sep 27 2018, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T09:54:43Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T205514: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 56s)

• Banyek added a comment.Sep 27 2018, 12:12 PM

recloning finished, the hosts are replicating again

• Banyek moved this task from next to In progress on the User-Banyek board.Sep 27 2018, 1:31 PM
Stashbot added a comment.Sep 27 2018, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T13:41:32Z] <banyek> repooling db1104 (T205514)

Stashbot added a comment.Sep 27 2018, 1:46 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T13:46:23Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T205514: revert: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 55s)

Stashbot added a comment.Sep 27 2018, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-27T13:49:54Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T205514: revert: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 56s)

• Banyek moved this task from In progress to Wait on external on the User-Banyek board.Sep 27 2018, 1:51 PM
Cmjohnson added a comment.Sep 27 2018, 5:06 PM

the HP required AHS log has been uploaded to their dropbox. Waiting on their response.

Cmjohnson added a comment.Oct 2 2018, 4:42 PM

Waiting on the part still

gerritbot added a comment.Oct 5 2018, 5:08 AM

Change 464753 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Clarify db1092 status

https://gerrit.wikimedia.org/r/464753

gerritbot added a comment.Oct 5 2018, 5:12 AM

Change 464753 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Clarify db1092 status

https://gerrit.wikimedia.org/r/464753

Stashbot added a comment.Oct 5 2018, 5:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-05T05:13:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1092 status - T205514 (duration: 00m 57s)

gerritbot added a comment.Oct 8 2018, 5:41 AM

Change 465120 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1092 with low weight

https://gerrit.wikimedia.org/r/465120

gerritbot added a comment.Oct 8 2018, 7:18 AM

Change 465120 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1092 with low weight

https://gerrit.wikimedia.org/r/465120

Stashbot added a comment.Oct 8 2018, 7:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-08T07:20:33Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1092 with low weight - T205514 (duration: 01m 27s)

Cmjohnson added a comment.Oct 9 2018, 5:32 PM

The battery was sent to our old office address in San Francisco, they are shipping a new battery...because it's a battery it has to go ground and will be 3-5 days

Marostegui added a comment.Oct 10 2018, 5:12 AM

Thanks for the update Chris - unbelievable!

gerritbot added a comment.Oct 11 2018, 5:53 AM

Change 465959 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1092

https://gerrit.wikimedia.org/r/465959

gerritbot added a comment.Oct 11 2018, 5:59 AM

Change 465959 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1092

https://gerrit.wikimedia.org/r/465959

Stashbot added a comment.Oct 15 2018, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-15T15:39:03Z] <marostegui> Stop MySQL and poweroff db1092 for BBU replacement - T205514

Marostegui closed this task as Resolved.Oct 15 2018, 3:55 PM

Battery replaced by Chris - thank you!:

Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL