A cursory look shows two standing problems that are related and possibly blocking:

1. it cannot handle a configuration where a host has both mixed RAID and standalone disks (Note: mdraid is considered standalone disks)
2. it cannot emit logs that aren't high noise as they go directly to cronspam

Given the size of both of these problems, they likely merit their own tasks.

A short term solution occurs to me. What if on each run it exported the number of disks it has detected. It's seems reasonable to assume that a host indicating it has no disks is misbehaving in some way.

I think it is fair to say that if no disks are detected then that's always an error condition (?) In that case I think a simple(r) solution would be to exit non-zero if no disks are detected so the systemd service/timer fails loudly.

The related problem to this one is also that the hp raid controller name changed when we changed the facter invocation, and ssacli isn't recognized as such by smart_data_dump

In T267135#6615766, @fgiunchedi wrote:

The related problem to this one is also that the hp raid controller name changed when we changed the facter invocation, and ssacli isn't recognized as such by smart_data_dump

Upon further investigation this doesn't seem to correct, as in I'm getting ssacli as raid fact with either invocation. The underlying issue though remains :|

root@ms-be1056:~# facter --puppet --json -l error raid
{
  "raid": [
    "md",
    "ssacli"
  ]
}
root@ms-be1056:~# /usr/bin/ruby /var/lib/puppet/lib/facter/raid.rb
{"raid":["md","ssacli"]}

Per discussion on IRC, we know two things:

1. The proposed change to exit non-zero when no disks are detected is very likely to be extremely noisy given we do not know the extent of the problem. The noise generated is on the order of (affected hosts)x(check frequency).
2. The fix is non-trivial given that it requires adding a long-requested feature to enable collection of smart metrics for hosts containing both standalone disks and raid. This necessitates some form of deduplication subroutine to detect which disks are handled by the raid controller, which are not, and query them as appropriate for the way the disks are attached to the host. It also necessitates onboarding at least one new raid controller type (ssacli).

Given this situation, we will:

1. Add a metric which counts the number of disks detected by smart_data_dump
2. Improve smart_data_dump in response to the hosts currently affected
3. Add this feature so that future hosts indicate the problem early

Change 640473 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] smart: add metric to track number of devices detected

https://gerrit.wikimedia.org/r/640473

Change 640473 merged by Cwhite:
[operations/puppet@production] smart: add metric to track number of devices detected

https://gerrit.wikimedia.org/r/640473

25 hosts affected

device_smart_device_count < 1