db1075 (s3) primary master crashed due to BBU failure (T233535) and caused all the s3 wikis to be on read-only from 18:48 UTC to around 19:15 UTC
Reads were not affected
HW logs:
/system1/log1/record13 Targets Properties number=13 severity=Caution date=09/22/2019 time=18:37 description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support Verbs cd version exit show </system1/log1>hpiLO-> show record14 status=0 status_tag=COMMAND COMPLETED Mon Sep 23 05:01:26 2019 /system1/log1/record14 Targets Properties number=14 severity=Caution date=09/22/2019 time=19:00 description=POST Error: 313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists. Verbs cd version exit show
I rebooted the host and after a few checks I started mysql again
This host is scheduled to be failed over on Tuesday 24th - T230783: Switchover s3 primary database master db1075 -> db1123 - 24th Sept @05:00 UTC
A new BBU for this host should be bought: {T233567}
A BBU failure has resulted on hosts crashing entirely before T225391: db1077 crashed T231638: db1074 crashed: Broken BBU
This host is part of a 6 hosts batch {T118174} T128753: Rack and Initial setup db1074-79 and 3 of them have had BBU failures.