db1021 experienced sudden replag, which gdb showed was threads writing to disk
blocking on fsync(). Digging around:
megacli -AdpBbuCmd -a0 | grep Relative
Relative State of Charge: 9 %
And BBU charge was climbing, implying it was lower, perhaps even 0%? Looks like
the write-back cache was disabled in the process.
megacli -AdpBbuCmd -GetBbuProperties -a0 | grep "Learn"
Auto Learn Period: 90 Days
Next Learn time: Wed Oct 15 12:04:59 2014
Learn Delay Interval:0 Hours
Auto-Learn Mode: Enabled
90 days before that is right now. Investigate disabling auto-learn on db hosts.
Description
Description
Details
Details
- Reference
- rt7916
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Create new module for managing RAID settings | operations/puppet | production | +58 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T84178 investigate RAID BBU auto-learn on db hosts | |||
| Resolved | faidon | T84050 Refactor RAID checks (check-raid) | |||
| Open | None | T83476 Icinga RAID check: monitor rebuild status | |||
| Resolved | faidon | T97998 Add RAID monitoring for HP servers | |||
| Resolved | herron | T141252 icinga hp raid check timeout on busy ms-be and db machines | |||
| Resolved | herron | T172921 Nrpe command_timeout and "Service Check Timed Out" errors |
Event Timeline
Comment Actions
Consider using "Write Cache OK if Bad BBU" to avoid dropping to WiteThrough
during learning. Would require more monitoring including battery state.
Comment Actions
Yes, this would definitely do it, I've heard about this before.
Note that not too long ago at #7780 I wrote "or even weird statuses such as battery train schedules", referring exactly to something like this.
Faidon
Comment Actions
This is made more complicated by the switch to HP boxes in CODFW. May be related to, or influenced by, T97998.
Comment Actions
I do not feel comfortable with enforcing "Write Cache OK if Bad BBU", specially with the current status of RAID monitoring.
My proposed solution right now is:
- Disable the automatic learning phase so that it only warns every 3 months:
echo "autoLearnMode=2" > BbuProperties
megacli -AdpBbuCmd -SetBbuProperties -f BbuProperties -a0
(doesn't work on all RAIDs, we will have to disable it on some "autoLearnMode=1").
Worst case scenario: no learn cycle is ever done instead of: actual hardware defect is not detected
From the manual:
"Warn Via Event: The firmware warns about a pending learning cycle. You can initiate a learning cycle manually.
After the learning cycle is complete, the firmware resets the counter and warns you when the next learning cycle
time is reached."
- Schedule maintenance manually making sure it includes a manual learning phase: $ megacli -AdpBbuCmd -BbuLearn -a0. This should be done on T84050 by detecting the RAID warning and, and a warning is sent to icinga.
When battery checks are in place (aside from being WIP, most the RAID-using hosts are not checking it on nagios), add that RAID alert to nagios if necessary.
This comment was removed by jcrespo.
Comment Actions
All database hosts that allow "Warn only" have been setup as such. On the few that didn't, it has been disabled. This is the script that has been run:
salt '<hosts>' cmd.run 'echo "autoLearnMode=1" > /tmp/BbuProperties; megacli -AdpBbuCmd -SetBbuProperties -f /tmp/BbuProperties -a0; echo "autoLearnMode=2" > /tmp/BbuProperties; megacli -AdpBbuCmd -SetBbuProperties -f /tmp/BbuProperties -a0; rm /tmp/BbuProperties'
This should fix immediate issues, and maybe we can revisit in the future better monitoring or a plan for planned BBU maintenance.