Page MenuHomePhabricator
Create Task
Maniphest T222127

db1093 (s6 candidate master) went down - broken BBU
Closed, ResolvedPublic
Actions

Description

dba1093 went down today:

19:24 <+icinga-wm> PROBLEM - Host db1093 is DOWN: PING CRITICAL - Packet loss = 100%

later it came back up:

21:44 <+icinga-wm> RECOVERY - Host db1093 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms

at this point we got paged about it because

21:46 <+icinga-wm> PROBLEM - mysqld processes on db1093 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting

We figured that it's normal and as desired that mysql does not come up after reboots.. but that we are still supposed to depool it.

Followed the docs at https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_slave

and made: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/507237

which tgr kindly deployed

CRITICAL: Slot 1: OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 2I:2:1, 2I:2:2 - Controller: OK - Battery count: 0

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Fully repool db1093operations/mediawiki-configmaster+1 -1
db-eqiad.php: Depoool db1093operations/mediawiki-configmaster+2 -2
db-eqiad.php: Give some API weight to db1093operations/mediawiki-configmaster+1 -1
db-eqiad.php: Give some traffic to db1093operations/mediawiki-configmaster+1 -1
db1093: Enable notificationsoperations/puppetproduction+0 -1
db-eqiad.php: Clarify db1093 statusoperations/mediawiki-configmaster+1 -1
db1093: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Dzahn created this task.Apr 30 2019, 2:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2019, 2:10 AM
Dzahn added a comment.Apr 30 2019, 2:12 AM

also about a minute later:

22:10 <+icinga-wm> PROBLEM - HP RAID on db1093 is CRITICAL: CRITICAL: Slot 1: OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 2I:2:1, 2I:2:2 - Controller: OK - Battery count: 0

which automatically created T222128

22:10 < mutante> and that RAID issue is why it went down.. i think
22:10 < mutante> there were lots of alerts in SOFT state earlier
22:11 < mutante> now it finally went from SOFT to HARD

Dzahn added a comment.Apr 30 2019, 2:12 AM

could confirm the changes appeared on https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php

Dzahn mentioned this in T222128: Degraded RAID on db1093.Apr 30 2019, 2:13 AM
gerritbot added a comment.Apr 30 2019, 5:04 AM

Change 507241 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1093: Disable notifications

https://gerrit.wikimedia.org/r/507241

gerritbot added a project: Patch-For-Review.Apr 30 2019, 5:04 AM
• Marostegui merged a task: T222128: Degraded RAID on db1093.Apr 30 2019, 5:04 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)
• Marostegui added subscribers: ops-monitoring-bot, • Marostegui.
• Marostegui added a subscriber: Tgr.

Thanks a lot @Dzahn and @Tgr for taking care of this - we will take it from here
Much appreciated

gerritbot added a comment.Apr 30 2019, 5:08 AM

Change 507241 merged by Marostegui:
[operations/puppet@production] db1093: Disable notifications

https://gerrit.wikimedia.org/r/507241

• Marostegui renamed this task from db1093 went down - depooled to db1093 (s6 candidate master) went down - broken BBU.Apr 30 2019, 5:16 AM
• Marostegui assigned this task to • Cmjohnson.
• Marostegui added a subscriber: • Cmjohnson.

The BBU looks broken:

/system1/log1/record13
  Targets
  Properties
    number=13
    severity=Caution
    date=04/29/2019
    time=23:19
    description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support
  Verbs
    cd version exit show

/system1/log1/record14
  Targets
  Properties
    number=14
    severity=Critical
    date=04/30/2019
    time=01:41
    description=ASR Detected by System ROM
  Verbs
    cd version exit show

/system1/log1/record15
  Targets
  Properties
    number=15
    severity=Caution
    date=04/30/2019
    time=01:42
    description=POST Error: 313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists.
  Verbs
    cd version exit show
root@db1093:~# hpssacli controller all show detail | grep -i battery
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0

@Cmjohnson can we contact HP to get a new BBU?
Also, upon installation of the new BBU, we should make sure we are running the latest BIOS version and firmware.
Triaging this as HIGH because it is the candidate master

• Marostegui triaged this task as High priority.Apr 30 2019, 5:17 AM
• Marostegui added a project: ops-eqiad.
Restricted Application added a project: SRE. · View Herald TranscriptApr 30 2019, 5:18 AM
gerritbot added a comment.Apr 30 2019, 5:22 AM

Change 507242 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Clarify db1093 status

https://gerrit.wikimedia.org/r/507242

gerritbot added a comment.Apr 30 2019, 5:25 AM

Change 507242 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Clarify db1093 status

https://gerrit.wikimedia.org/r/507242

• Marostegui added a comment.Apr 30 2019, 5:31 AM

I have started MySQL which started correctly.
As it started fine, I have started replication too, once it has caught up, I am going to do a data check with another host to make sure the data isn't corrupted.

• Marostegui moved this task from Triage to In progress on the DBA board.Apr 30 2019, 6:19 AM
• Marostegui added a comment.Apr 30 2019, 6:57 AM

For what is worth, the LB looks like it worked fine.
The time line is:

23:24: db1093 goes down
23:24-23:30: Spike of errors and then some residual ones from jobrunner mostly: https://logstash.wikimedia.org/goto/0afd44275cc38db84317bfb02257ebe0
23:30: no more errors
02:09: db1093 is depooled

• Marostegui added a comment.Apr 30 2019, 9:02 AM

The following tables have been checked against multiple hosts and reported no differences:

archive
logging
page
revision
text
user
change_tag
actor
ipblocks
comment

So I am going to repool this host with low weight as it doesn't have BBU, so it just receive some queries at least.

gerritbot added a comment.Apr 30 2019, 9:11 AM

Change 507263 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Give some traffic to db1093

https://gerrit.wikimedia.org/r/507263

gerritbot added a comment.Apr 30 2019, 9:13 AM

Change 507264 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1093: Enable notifications

https://gerrit.wikimedia.org/r/507264

Dzahn moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Apr 30 2019, 6:33 PM
gerritbot added a comment.May 6 2019, 5:02 AM

Change 507264 merged by Marostegui:
[operations/puppet@production] db1093: Enable notifications

https://gerrit.wikimedia.org/r/507264

gerritbot added a comment.May 6 2019, 5:04 AM

Change 507263 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Give some traffic to db1093

https://gerrit.wikimedia.org/r/507263

gerritbot added a comment.May 6 2019, 5:53 AM

Change 508167 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Give some API weight to db1093

https://gerrit.wikimedia.org/r/508167

gerritbot added a comment.May 6 2019, 6:00 AM

Change 508167 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Give some API weight to db1093

https://gerrit.wikimedia.org/r/508167

RobH subscribed.May 6 2019, 4:13 PM

This is only under warranty until later this month, and was brought up in the SRE weekly meeting. This needs to be high priority! Supposedly warranty is until May 24th in netbox, but please treat this as top action item!

• Cmjohnson added a comment.May 6 2019, 4:31 PM

I created a task for this with HPE.

Case ID: 5338390467
Case title:
Failed BBU
Severity 3-Normal
Product serial number: MXQ616071T
Product number: 755258-B21
Submitted: 5/6/2019 12:30:17 PM
Last updated: 5/6/2019 12:30:17 PM
Source: Web
Case status: Received by HPE

gerritbot added a comment.May 7 2019, 1:42 PM

Change 508574 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depoool db1093

https://gerrit.wikimedia.org/r/508574

gerritbot added a comment.May 7 2019, 1:44 PM

Change 508574 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depoool db1093

https://gerrit.wikimedia.org/r/508574

Stashbot added a comment.May 7 2019, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T13:45:26Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1093 for BBU replacement T222127 (duration: 00m 51s)

Stashbot added a comment.May 7 2019, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T13:45:32Z] <marostegui> Stop MySQL and poweroff db1093 for BBU replacement - T222127

• Marostegui added a comment.May 7 2019, 1:58 PM

Chris has changed the BBU and I can already see it:

root@db1093:~#  hpssacli controller all show detail | grep -i battery
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK

I have started MySQL but won't repool the host till tomorrow, to give it sometime to make sure it is all fine.

gerritbot added a comment.May 8 2019, 8:16 AM

Change 508780 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1093

https://gerrit.wikimedia.org/r/508780

gerritbot added a comment.May 8 2019, 8:28 AM

Change 508780 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1093

https://gerrit.wikimedia.org/r/508780

• Marostegui closed this task as Resolved.May 8 2019, 8:30 AM

This host has been fully repooled
Thanks @Cmjohnson for replacing the BBU

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL