during disk-intensive operations (e.g. swift rebalance or mysql alter table) we've seen the check_hpssacli nrpe check timing out on newer hp machines. The timeout has been increased to 40s already but sometimes that doesn't seem sufficient.
Swift machines are a bit peculiar in which they have all disks in raid0, so for each disk there's a physical volume and a logical volume to be checked.
MySQL is not peculiar and that also happens when there is high io_activity.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T294906 Puppet Improvements | |||
| Duplicate | jbond | T265138 Work required to prepare for puppet 7 | |||
| Resolved | SLyngshede-WMF | T273673 replace all puppet crons with systemd timers | |||
| Open | None | T132324 Tracking and Reducing cron-spam to root@ | |||
| Resolved | jcrespo | T84178 investigate RAID BBU auto-learn on db hosts | |||
| Resolved | faidon | T84050 Refactor RAID checks (check-raid) | |||
| Resolved | faidon | T97998 Add RAID monitoring for HP servers | |||
| Resolved | herron | T141252 icinga hp raid check timeout on busy ms-be and db machines | |||
| Resolved | herron | T172921 Nrpe command_timeout and "Service Check Timed Out" errors |
I've checked, and the currently in use check does too much, probably we do not need such a thorough check every time icinga runs, which would solve the issues with the timeout. Just returning a full output of the configuration and status is not that heavy, so we may be able to compromise here.
Sure, sounds good to me. We could always reopen and evaluate if the issue occurs again in the future.