Icinga notifications didn't get applied after a puppet run
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Marostegui
	Apr 29 2020, 12:47 PM

Description

The following change: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593205/ got merged at around 10:39 AM UTC
It disables notification on db2087.
Puppet on icinga that it indeed worked:

Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) --- /etc/icinga/objects/puppet_hosts.cfg#0112020-04-29 07:46:38.198594904 +0000
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) +++ /tmp/puppet-file20200429-180139-qd2969#0112020-04-29 11:16:26.606831836 +0000
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) @@ -11299,7 +11299,7 @@
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)  #011notification_interval          0
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)  #011notification_options           d,u,r,f
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)  #011parents                        asw-c-codfw
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)  #011statusmap_image                vendors/debian.gd2
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content)
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) content changed '{md5}85a17ee73acef493ae34f138bdca828c' to '{md5}66f324205c205d22b8292ce3cf57e7e2'
Apr 29 11:16:26 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]) Scheduling refresh of Service[icinga]
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) --- /etc/icinga/objects/puppet_services.cfg#0112020-04-29 10:17:05.887931247 +0000
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +++ /tmp/puppet-file20200429-180139-1x3aof#0112020-04-29 11:16:27.106833915 +0000
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256431,7 +256431,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            dhclient process
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256453,7 +256453,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            configured eth
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256475,7 +256475,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            IPMI Sensor Status
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256497,7 +256497,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Check systemd state
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256519,7 +256519,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Check size of conntrack table
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256541,7 +256541,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Check whether microcode mitigations for CPU vulnerabilities are applied
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256563,7 +256563,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Disk space
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256607,7 +256607,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            DPKG
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256629,7 +256629,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Memory correctable errors -EDAC-
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256651,7 +256651,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            EDAC syslog messages
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256673,7 +256673,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Check whether ferm is active by checking the default input chain
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256695,7 +256695,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Filesystem available is greater than filesystem size
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256717,7 +256717,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB disk space
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256739,7 +256739,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB read only s6
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256761,7 +256761,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB read only s7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256783,7 +256783,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave IO: s6
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256805,7 +256805,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave IO: s7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256827,7 +256827,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave Lag: s6
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256849,7 +256849,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave Lag: s7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256871,7 +256871,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave SQL: s6
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256893,7 +256893,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MariaDB Slave SQL: s7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256915,7 +256915,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            mysqld processes
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256937,7 +256937,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            puppet last run
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256960,7 +256960,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            MegaRAID
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -256982,7 +256982,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Device not healthy -SMART-
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -257004,7 +257004,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            SSH
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -257048,7 +257048,7 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) -#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011service_description            Check the NTP synchronisation status of timesyncd
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -411978,6 +411978,50 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  }
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  define service {
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +# --PUPPET_NAME-- idp-test1001 envoy_runtime_vars
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011active_checks_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_command                  nrpe_check!check_envoy_runtime_vars!10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_freshness                0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_interval                 30
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_period                   24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011contact_groups                 admins
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011host_name                      idp-test1001
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011is_volatile                    0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011max_check_attempts             2
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notes_url                      https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Envoy
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011retry_interval                 5
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011service_description            Check no envoy runtime configuration is left persistent
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011servicegroups                  misc_eqiad
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +}
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +define service {
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +# --PUPPET_NAME-- idp-test1001 envoyproxy.service-state
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011active_checks_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_command                  nrpe_check!check_envoyproxy.service-state!10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_freshness                0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_period                   24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011contact_groups                 admins
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011host_name                      idp-test1001
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011is_volatile                    0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011max_check_attempts             2
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notes_url                      https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Envoy
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011service_description            Check that envoy is running
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011servicegroups                  misc_eqiad
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +}
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +define service {
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  # --PUPPET_NAME-- idp-test1001 ferm_active
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011active_checks_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011check_command                  nrpe_check!check_ferm_active!10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) @@ -412131,6 +412175,28 @@
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011servicegroups                  misc_codfw
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  }
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +define service {
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +# --PUPPET_NAME-- idp-test2001 check_idp-u2f-sync_status
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011active_checks_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_command                  nrpe_check!check_check_idp-u2f-sync_status!10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_freshness                0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_interval                 10
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011check_period                   24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011contact_groups                 admins
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011host_name                      idp-test2001
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011is_volatile                    0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011max_check_attempts             2
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notes_url                      https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_interval          0
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_options           c,r,f
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notification_period            24x7
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011notifications_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011passive_checks_enabled         1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011retry_interval                 1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011service_description            Check the last execution of idp-u2f-sync
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +#011servicegroups                  misc_codfw
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) +}
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  define service {
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  # --PUPPET_NAME-- idp-test2001 check_long_procs
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content)  #011active_checks_enabled          1
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]/content) content changed '{md5}1ddff12c42d183b79b6096599c00ca36' to '{md5}815c8d203b6ed90881abd8487bc69114'
Apr 29 11:16:27 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_services.cfg]) Scheduling refresh of Service[icinga]
Apr 29 11:17:05 icinga1001 puppet-agent[180139]: (/Stage[main]/Icinga/Systemd::Service[icinga]/Service[icinga]) Triggered 'refresh' from 2 events
Apr 29 11:17:08 icinga1001 puppet-agent[180139]: Applied catalog in 65.07 seconds

However, the host alerted and icinga's UI confirmed that not all the checks got disabled: P11083#63751

And puppet agent ran on the host several times after the above changed was applied on Icinga:

Apr 29 11:36:18 db2087 puppet-agent[11262]: Using configured environment 'production'
Apr 29 11:36:18 db2087 puppet-agent[11262]: Retrieving pluginfacts
Apr 29 11:36:18 db2087 puppet-agent[11262]: Retrieving plugin
Apr 29 11:36:18 db2087 puppet-agent[11262]: Retrieving locales
Apr 29 11:36:18 db2087 puppet-agent[11262]: Loading facts
Apr 29 11:36:28 db2087 puppet-agent[11262]: Caching catalog for db2087.codfw.wmnet
Apr 29 11:36:29 db2087 puppet-agent[11262]: Applying configuration version '(d33cdaa8cd) Kormat - install_server: switch db2087 to buster'
Apr 29 12:06:48 db2087 puppet-agent[16664]: Using configured environment 'production'
Apr 29 12:06:48 db2087 puppet-agent[16664]: Retrieving pluginfacts
Apr 29 12:06:48 db2087 puppet-agent[16664]: Retrieving plugin
Apr 29 12:06:49 db2087 puppet-agent[16664]: Retrieving locales
Apr 29 12:06:49 db2087 puppet-agent[16664]: Loading facts
Apr 29 12:06:58 db2087 puppet-agent[16664]: Caching catalog for db2087.codfw.wmnet
Apr 29 12:06:59 db2087 puppet-agent[16664]: Applying configuration version '(9e7fc4aeeb) Ema - vcl: move 'exp' admission policy to wm_admission_policies'

This is when the host alerted:

[12:33:31]  <+icinga-wm>	PROBLEM - MariaDB Slave IO: s6 on db2087 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave
[12:33:33]  <+icinga-wm>	PROBLEM - MariaDB Slave SQL: s7 on db2087 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave
[12:33:49]  <+icinga-wm>	PROBLEM - mysqld processes on db2087 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting

Related Objects

Mentioned Here: P11083 (An Untitled Masterwork)

Event Timeline

Marostegui created this task.Apr 29 2020, 12:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2020, 12:47 PM

Marostegui updated the task description. (Show Details)Apr 29 2020, 12:51 PM

Interestingly, after the host was reimaged and dropped from icinga...when it got created again, it did have all the notifications disabled.

It's notable that:

now all services on db2087 DO have disabled notifications
in the pasted log above the "host_name" parameter only shows up for "idp-*" hosts but does not really show db2087
host was reimaged
in the screenshot pasted above only some services have disabled notifications but all services are in downtime. and being in downtime should also mean no notifications despite the additional disabling discussed here

In T251407#6093444, @Dzahn wrote:

It's notable that:

in the screenshot pasted above only some services have disabled notifications but all services are in downtime. and being in downtime should also mean no notifications despite the additional disabling discussed here

The downtime was done after MySQL was stopped.

MySQL gets stopped and icinga starts to get the failures counter
Reimage starts but the counter is already almost done, so the notification gets sent.

From my experience puppet has to run first on the monitored host and afterwards on icinga itself and likely more than once. Maybe it ran once and then the host went down for reimaging? That's what it seems like but it doesn't add up that there were 2 hours in between.

Puppet ran at least twice on the host itself and it did on Icinga as well, likes it ran 3 times on the icinga host between the first disablement and right before the host was sent for reimage:

Apr 29 11:17:08 icinga1001 puppet-agent[180139]: Applied catalog in 65.07 seconds
Apr 29 11:46:56 icinga1001 puppet-agent[163898]: Applied catalog in 58.72 seconds
Apr 29 12:16:53 icinga1001 puppet-agent[183051]: Applied catalog in 58.19 seconds

colewhite triaged this task as Medium priority.May 4 2020, 11:04 PM

This just happened again with https://gerrit.wikimedia.org/r/c/operations/puppet/+/602014

I manually ran puppet on the target host, and confirmed that puppet had run on icinga before i stopped mariadb on the target host. Icinga still alerted.

The web u/i has finally updated to show that _some_ services have notifications disabled, but not all.

per chat on IRC: puppet was already running on icinga1001 when the puppet run on the monitored host was started. Because of exported resources a run is needed on the monitored host followed by one on icinga itself.

A little while later (during reimaging) pc2009 was removed from Icinga and then re-added with PENDING checks, downtimes and all notifications disabled.

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 20 2020, 1:11 PM

lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

Maintenance_bot added a project: observability.Jul 12 2021, 2:50 AM

lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM

lmata edited projects, added Observability-Alerting; removed SRE Observability.Aug 9 2021, 3:43 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Alerting.Aug 9 2021, 3:49 AM

lmata edited projects, added Observability-Alerting; removed SRE Observability.Jan 31 2022, 1:46 PM

	F31851983: Screenshot at 2020-06-03 05-36-55.png
	Jun 3 2020, 9:38 AM

	F31851959: image.png
	Jun 3 2020, 9:03 AM

Icinga notifications didn't get applied after a puppet runOpen, MediumPublicActions

Description

Related Objects

Event Timeline

Icinga notifications didn't get applied after a puppet run
Open, MediumPublic
Actions