Due to db1078's crash (T209754) I had to disable notifications for that host.
I did it via puppet: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474447/
I ran puppet on db1078 and after that I ran puppet on icinga1001:
```
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for icinga1001.wikimedia.org
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1542437455'
Notice: /Stage[main]/Icinga/File[/var/lib/icinga/retention.dat]/group: group changed 'nagios' to 'www-data'
Notice: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content:
--- /etc/icinga/objects/puppet_hosts.cfg 2018-11-16 21:45:49.741852752 +0000
+++ /tmp/puppet-file20181117-48036-zhxmnc 2018-11-17 06:52:09.977243985 +0000
@@ -8377,7 +8377,7 @@
notification_interval 0
notification_options d,u,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
parents asw2-c-eqiad
statusmap_image vendors/debian.gd2
Notice: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content: content changed '{md5}e0cf8894edd8e2d2374ff536fc2d255d' to '{md5}457fc8ebdad17df6e9308dd06bd5e82e'
Info: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]: Scheduling refresh of Service[icinga]
Notice: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content:
--- /etc/icinga/objects/puppet_services.cfg 2018-11-16 21:45:50.173853762 +0000
+++ /tmp/puppet-file20181117-48036-1283is9 2018-11-17 06:52:10.237244572 +0000
@@ -159730,7 +159730,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description dhclient process
@@ -159751,7 +159751,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description configured eth
@@ -159772,7 +159772,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 10
service_description IPMI Sensor Status
@@ -159793,7 +159793,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description Check systemd state
@@ -159814,7 +159814,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description Check size of conntrack table
@@ -159835,7 +159835,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description Disk space
@@ -159877,7 +159877,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description DPKG
@@ -159899,7 +159899,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 5
service_description Memory correctable errors -EDAC-
@@ -159920,7 +159920,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description Check whether ferm is active by checking the default input chain
@@ -159942,7 +159942,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 5
service_description Filesystem available is greater than filesystem size
@@ -159963,7 +159963,7 @@
notification_interval 240
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description MariaDB disk space
@@ -159984,7 +159984,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description MariaDB read only s3
@@ -160005,7 +160005,7 @@
notification_interval 240
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description MariaDB Slave IO: s3
@@ -160026,7 +160026,7 @@
notification_interval 240
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description MariaDB Slave Lag: s3
@@ -160047,7 +160047,7 @@
notification_interval 240
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description MariaDB Slave SQL: s3
@@ -160068,7 +160068,7 @@
notification_interval 240
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description mysqld processes
@@ -160089,7 +160089,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description puppet last run
@@ -160111,7 +160111,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 10
service_description HP RAID
@@ -160133,7 +160133,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 5
service_description Device not healthy -SMART-
@@ -160154,7 +160154,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description SSH
@@ -160196,7 +160196,7 @@
notification_interval 0
notification_options c,r,f
notification_period 24x7
- notifications_enabled 1
+ notifications_enabled 0
passive_checks_enabled 1
retry_interval 1
service_description Check the NTP synchronisation status of timesyncd
Notice: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content: content changed '{md5}9a3a82240e9d4e60cee10a1deb89a6e6' to '{md5}ab420d7c1a60aa1966f2ab12d2e9b403'
Info: /Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]: Scheduling refresh of Service[icinga]
Notice: /Stage[main]/Icinga::Web/Letsencrypt::Cert::Integrated[icinga]/Exec[acme-setup-acme-icinga]/returns: executed successfully
Notice: /Stage[main]/Icinga/Systemd::Service[icinga]/Service[icinga]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 65.78 seconds
```
However, on Icinga's UI I only saw _some_ services being disabled.
This is the IRC conversation that happened after me asking:
```
06:56 < marostegui> shdubsh: I pushed: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474447/ but I am only seeing a few services disabled, not all of them, is that a known icigan issue?
06:58 < volans|off> marostegui: did puppet run first on db1078 and then on icinga? I'm checkking anyway
06:59 < marostegui> I did manually first on db1078 and after that on icinga
06:59 < volans|off> which icinga?
06:59 < volans|off> icinga1001, not einsteinium
06:59 < marostegui> volans|off: 1001
06:59 < volans|off> interesting... checking
07:00 < volans|off> I can see some of them have a local modification, let me reset all the local states for all services on that host
07:03 < shdubsh> marostegui: not a known issue, no. That code path should be intact as we tried to avoid touching base as much as possible
07:03 < volans|off> the notifications_enabled seems to be 0 in the config as expected
07:04 < marostegui> volans|off: yeah, I saw that on puppet
07:06 < volans|off> and /var/cache/icinga/objects.cache too have the correct value (0)
07:07 < volans|off> shdubsh: btw that file was -rw-r--r-- 1 nagios nagios, after your patch is -rw-r--r-- 1 nagios www-data
07:07 < volans|off> [totally unrelated, just noticed]
07:10 < shdubsh> That's expected. The permissions were copied from a clean install
07:10 < volans|off> shdubsh: both status.dat and retention.dat have notifications_enabled=1
```