Status	Assigned	Task
		Unknown Object (Task)
Resolved	Marostegui	T258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
Declined	None	T258386 db1080-95 batch possibly suffering BBU issues
Resolved	Marostegui	T258360 db1085 crashed

Event Timeline

CDanis created this task.Jul 19 2020, 6:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 19 2020, 6:46 PM

CDanis triaged this task as High priority.Jul 19 2020, 6:47 PM

BBU issues as expected. This host is also scheduled to be refreshed next Q:

/system1/log1/record16
  Targets
  Properties
    number=16
    severity=Caution
    date=07/19/2020
    time=18:44
    description=POST Error: 313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists.
  Verbs
    cd version exit show


</system1/log1>hpiLO-> show record15

status=0
status_tag=COMMAND COMPLETED
Sun Jul 19 18:48:08 2020



/system1/log1/record15
  Targets
  Properties
    number=15
    severity=Critical
    date=07/19/2020
    time=18:43
    description=ASR Detected by System ROM
  Verbs
    cd version exit show


</system1/log1>hpiLO-> show record14

status=0
status_tag=COMMAND COMPLETED
Sun Jul 19 18:48:11 2020



/system1/log1/record14
  Targets
  Properties
    number=14
    severity=Caution
    date=07/19/2020
    time=18:26
    description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support
  Verbs
    cd version exit show

Pretty much the same issue as T258336

Marostegui mentioned this in T258361: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers).Jul 19 2020, 6:54 PM

And the BBU is gone:

root@db1085:~#  hpssacli controller all show detail | grep -i Battery
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0

Change 614590 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1085: Disable notifications

https://gerrit.wikimedia.org/r/614590

Change 614590 merged by Marostegui:
[operations/puppet@production] db1085: Disable notifications

https://gerrit.wikimedia.org/r/614590

Mentioned in SAL (#wikimedia-operations) [2020-07-19T19:16:10Z] <marostegui> Upgrade and reboot db1085 T258360

Marostegui mentioned this in T258362: Degraded RAID on db1085.Jul 19 2020, 7:16 PM

Marostegui merged a task: T258362: Degraded RAID on db1085.

Marostegui added a subscriber: ops-monitoring-bot.

Marostegui updated the task description. (Show Details)Jul 19 2020, 7:20 PM

Host upgraded and rebooted.
MySQL looks ok, replication started

Maintenance_bot removed a project: Patch-For-Review.Jul 19 2020, 8:10 PM

Marostegui mentioned this in T258386: db1080-95 batch possibly suffering BBU issues.Jul 20 2020, 10:49 AM

Marostegui added a parent task: T258386: db1080-95 batch possibly suffering BBU issues.

Change 615165 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1085: Enable notifications

https://gerrit.wikimedia.org/r/615165

gerritbot added a project: Patch-For-Review.Jul 21 2020, 10:09 AM

Change 615165 merged by Marostegui:
[operations/puppet@production] db1085: Enable notifications

https://gerrit.wikimedia.org/r/615165

Mentioned in SAL (#wikimedia-operations) [2020-07-21T10:45:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11985 and previous config saved to /var/cache/conftool/dbconfig/20200721-104546-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-07-21T10:58:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11986 and previous config saved to /var/cache/conftool/dbconfig/20200721-105852-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-07-21T11:08:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11987 and previous config saved to /var/cache/conftool/dbconfig/20200721-110854-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jul 21 2020, 11:10 AM

I have fully repooled this host.
It doesn't have a BBU, but s6 doesn't really have much load, so it will probably be able to keep up with replication without issues.
Next follow up will be done at T258386 and/or T258361

db1085 crashedClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

db1085 crashed
Closed, ResolvedPublic
Actions

Related Objects
Search...