We have a bunch of test servers that have all the checks configured in Icinga and then a very long scheduled downtime with an optional disabled notification. Usually the same thing happen for to-be-decom servers.
I think that this is the wrong approach because in time Icinga base checks are added or renamed and the original scheduled downtime for the host and all services is now only for the host and some services, defying the purpose of having this host not alarming at any time.
We should instead puppetize this so that those kind of hosts have a special configuration in Icinga so that the check names have some sort of identifier like TEST INSTANCE or DECOM to clearly recognize them on Icinga and IRC and be configured so that they don't ever page.
Here some random examples, see the full list on Icinga downtimes page:
- [[ https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=cp1008 | cp1008 ]]: permanent temporary test host...
- Some labs servers, see T120047#2823179
- [[ https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=db1019 | db1019 ]]: decom
- [[ https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=db1073 | db1073 ]]: host with hardware issues, as of now 2 checks are not in scheduled downtime and notifications disabled