The production Icinga server (alerting_host) is einsteinium. The hardware needs to be replaced and it is running jessie.
This task consists of:
[x] provide replacement hardware, called icinga1001 [T201344]
[x] ensure the current alerting_host puppet role works on stretch, fix it if it doesn't (cloud VPS)
[x] make changes to the role so that it can be applied on the new host in eqiad without being the "active" host to avoid duplicate alerts and notification spam
[x] optional but very nice to have (otherwise we have to override jenkins style checks): refactor the icinga role to a profile, move the Hiera calls to parameters
[x] apply role on icinga1001, confirm puppet is fine, if not fix things
[x] confirm web UI works, test login, user privileges, sending email out as notification
[x] confirm all checks are green (or like in prod) , without having notifications enabled
[x] schedule switchover from einsteinium to icinga1001
[x] switch over and disable notifications permanently on einsteinium but leave it running for a grace period (?)