I have manually forced a BBU learn cycle and it is now looking fine:

root@db1048:~#  megacli -AdpBbuCmd -BbuLearn -aALL -NoLog

Adapter 0: BBU Learn Succeeded.

root@db1048:~# megacli -AdpBbuCmd  -a0

BBU status for Adapter: 0

BatteryType: BBU
Voltage: 4055 mV
Current: 166 mA
Temperature: 32 C
Battery State: Optimal
Relative State of Charge: 34 %
Charger Status: In Progress

root@db1048:~# megacli -ldinfo -l0 -a0 | grep Policy
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

The slave is catching up with no problems after the policy has changed to WB again:

Seconds_Behind_Master: 3756

And I have disabled the auto-learn cycle just in case:

root@db1048:~# echo "autoLearnMode=1" > disable_learn
root@db1048:~# megacli -AdpBbuCmd -a0 | grep Auto-Learn
  Auto-Learn Mode: Warn via Event
root@db1048:~# megacli -AdpBbuCmd -SetBbuProperties -f disable_learn -a0

Adapter 0: Set BBU Properties Succeeded.

Exit Code: 0x00
root@db1048:~#  megacli -AdpBbuCmd -a0 | grep Auto-Learn
  Auto-Learn Mode: Disabled

I will close this ticket for now, but at least we have it for the future, if we see this failing again on this host.

Do you think we should force a learning cycle to db1047 T159266 ?

I just tried - we will see!

But db1047 one has a different (and more worrying error) for BBU a1:

Battery State: Failed

db1047's BBU is acting weirdly
It goes from Failed -> Charging -> Failed
It is acting very weirdly, it has gone from

Relative State of Charge: 4 %
Charger Status: Unknown

To:

Relative State of Charge: 100 %
Charger Status: In Progress

But still marking it as Failed.
I will leave it like that and check it in a few hours.

Unfortunately, db1047's BBU looks totally broken, it is not making any sense in what it reports. Some places it says it is fully charged, some others don't, and it is always reporting FAILED.
It was worth trying but we will need to stick to T159266

I believe there was yesterday maintenance or trouble on Phabricator. I would ask RelEng first.

Yep, the deployment page said there was a phabricator update so maybe that put more stress on the server and made the BBU fail (again)?
Because the fact that the policy went back to "safe" mode twice already...

This has happened again:

˜/icinga-wm 17:47> PROBLEM - MariaDB Slave Lag: m3 on db1048 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 328.00 seconds

root@db1048:~#  megacli -ldinfo -l0 -a0 | grep Policy
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU

root@db1048:~# megacli -AdpBbuCmd  -a0

BBU status for Adapter: 0

BatteryType: BBU
Battery State: Unknown
  Battery backup charge time : 0 hours

BBU Capacity Info for Adapter: 0

  Relative State of Charge: 33 %
  Absolute State of charge: 3 %

Forced a relearn and the BBU updated its state:

root@db1048:~#  megacli -AdpBbuCmd -BbuLearn -aALL -NoLog

Adapter 0: BBU Learn Succeeded.

Exit Code: 0x00
root@db1048:~# megacli -AdpBbuCmd  -a0

BBU status for Adapter: 0

BatteryType: BBU
Voltage: 4029 mV
Current: 0 mA
Temperature: 33 C
Battery State: Degraded(Need Attention)

And it is now charging slowly compared to previous values:

root@db1048:~# megacli -AdpBbuCmd  -a0 | egrep "Relative|Absolute"
Relative State of Charge: 36 %
  Relative State of Charge: 36 %
  Absolute State of charge: 4 %

And it recovered:

root@db1048:~# megacli -AdpBbuCmd  -a0

BBU status for Adapter: 0

BatteryType: BBU
Voltage: 4058 mV
Current: 152 mA
Temperature: 33 C
Battery State: Optimal

root@db1048:~#  megacli -ldinfo -l0 -a0 | grep Policy
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

is there anything I need to be doing for this?

Do you have any spare BBU available?

@Marostegui yes, I can use one from a decommissioned server.

@Cmjohnson thanks. Let me coordinate this and we will arrange one day to do the swap.
@mmodell is there any problem if we take db1048 down for a few minutes to replace its faulty BBU? ( it is the phabricator slave, but as far as I know phabricator doesn't do master-slave query separation)

There are some reports running on the slave- We should point the slave to the master to avoid activity there thought the dns alias.

@Marostegui: correct, phabricator isn't currently querying the slave, other than the reports mentioned by @jcrespo.

Great, we can change the DNS and that's should be it! Thanks!

This just happened again:

root@db1048:~# megacli -AdpBbuCmd  -a0

BBU status for Adapter: 0

BatteryType: BBU
Battery State: Unknown
  Battery backup charge time : 0 hours

BBU Capacity Info for Adapter: 0

  Relative State of Charge: 26 %
  Absolute State of charge: 3 %
  Remaining Capacity: 43 mAh
  Full Charge Capacity: 167 mAh
  Run time to empty: Battery is not being charged.
  Average time to empty: Battery is not being charged.
  Estimated Time to full recharge: Battery is not being charged.

root@db1048:~#  megacli -ldinfo -l0 -a0 | grep Policy
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU

I have forced the relearn cycle.

I am going to get the dns changed so we can schedule a day to replace the BBU

Change 352769 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m3 slave to codfw master

https://gerrit.wikimedia.org/r/352769

And after the manual relearn it is back to normal state:

root@db1048:~#  megacli -ldinfo -l0 -a0 | grep Policy
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Change 352769 merged by Marostegui:
[operations/dns@master] wmnet: Point m3 slave to codfw master

https://gerrit.wikimedia.org/r/352769

@Marostegui Let me know if you want to do the bbu swap today?

Mentioned in SAL (#wikimedia-operations) [2017-05-09T14:09:33Z] <marostegui> Stop MySQL and shutdown db1048 (phabricator slave) to replace BBU - T160731

I am going to leave m3-slave pointing to the codfw master, until tomorrow just in case. If the host goes fine overnight, I will revert the change.

So far the lag is limited to 3~4 seconds according to tendril, while from Grafana is flat zero, maybe the dashboard is not graphing the right data?
See db1048 replication lag dashboard.

It was swapped a few weeks ago, but I guess the new one is also pretty old as it comes from hosts previously decommissioned - right @Cmjohnson ?

And db1048 returned to WriteBack policy less than 1h ago 😛

Same behaviour as we have seen before with faulty BBUs :-(

@Volans and @Marostegui I can do this as soon as you give me the word go but keep in mind this is only going to be temporary. the bbu's for this class and age of server are going out in record numbers. Please think about a replacement server sooner rather than later.

Thanks Chris, I will have this ready for tomorrow so we can do it tomorrow if that works for you?
We are aware that this will happen again, we are trying to get rid of most of the old hosts (those <db1050 - T134476). But it will take some time.

Great! ping when I can do the swap.

Change 356334 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m3 slave to codfw master

https://gerrit.wikimedia.org/r/356334

Change 356334 merged by Marostegui:
[operations/dns@master] wmnet: Point m3 slave to eqiad master

https://gerrit.wikimedia.org/r/356334

Mentioned in SAL (#wikimedia-operations) [2017-05-31T13:08:09Z] <marostegui> Stop MySQL on db1048 and shutdown the host for maintenance - T160731

@Cmjohnson db1048 is now down and ready for you to swap the BBU

Thanks!

replaced the battery with a well used one from a decom'd db. Hopefully this will work for long enough. Server has been powered on

Thanks Chris.
The battery is now charging

Battery State: Optimal
BBU Firmware Status:

  Charging Status              : Charging
  Relative State of Charge: 29 %
  Absolute State of charge: 9 %
  Remaining Capacity: 156 mAh
  Full Charge Capacity: 535 mAh
  Run time to empty: Battery is not being charged.
  Average time to empty: Battery is not being charged.
  Estimated Time to full recharge: 1 Hour, 43 Min.

I will revert the DNS patch tomorrow morning once the battery has recharged and all that.

The battery looks good now, it recharged, the temperature is ok and I have disabled the auto learn.
I have started MySQL and once it catches up I will merge te DNS revert

@Cmjohnson We are going to decom db1048 (but we are not ready yet), please do not take any action here, we will just clone it and ask you to unrack it. Opened for DBA tracking purposes only.

no worries, I was just moving it to a lower priority for me..I am couple of weeks away from tacking decom's

Not yet, this is still in use.