I get mail alerts from Shinken for Toolforge and they are currently too "volatile" to be of use. Most (?) of them complain of "PROBLEM alert - $host/Puppet staleness is WARNING", only to be followed up a short time later by "RECOVERY alert - $host/Puppet staleness is OK".

I didn't fix anything in the mean time, and I assume noone else did. And this is the problem: An alert is only helpful if it alerts you to do something. If you don't need to do something, these alerts just mask the real alerts that would require immediate intervention.

When investigating some alerts after the fact, they can often be linked to an OOM event or similar that caused Puppet to fail. But it would have been helpful if in that case the alert message would have been "$host is out of memory".

Unfortunately, I cannot offer a solution that just does the right thing™. But IMHO we should look for one that silences the constant beeping in the background, so that the "red alert!" stands out better.