Labservices1001 died (T152340) and I found out about it from the toolschecker. If the toolschecker had failed over properly between DNS hosts, we would not have know about this outage AT ALL until new VM creation started failing.
That's crazy. Why didn't a down host cause icinga to light up like a Christmas tree? Or, did it and I'm just not subscribed to those pages somehow?