Cleanup manually modified attributes on Icinga hosts and services
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Volans
	Thu, Sep 24, 8:27 AM

Project Tags

Referenced Files

None

Subscribers

Description

As a follow up of the related incident @tappof and I have been resetting the temporary disabled active checks and notifications from Icinga hosts and services that were disabled during the incident.
During this cleanup we've found also a series of services and checks that have modified attributes on the Icinga UI that don't come from Puppet and we would like to cleanup its state.

This is the extraction made today of the services with modified attributes:
356 other objects still have modified attributes (modified_attributes in status.dat: 1 = notifications, 2 = active checks, 3 = both). The mask by itself does not tell the direction of the change, if they should be enabled and are disabled or are disabled and should be enabled.

Mask	Hosts	Services
1	68	197
2	1	49
3	1	40

Cross checking that with the status.dat in Icinga this is what came out:

Notifications disabled at runtime
- ps1-d5-eqiad (host) and 6 infeed-load services.

Notifications enabled, but flagged as modified (probably disabled and then re-enabled by hand)
- 67 hosts: network devices (cr*, mr*, msw*, cloudsw*, fasw*, re0.*, scs-eqsin), ps1-* PDUs, en.wikipedia.org, en.m.wikipedia.org, lvs1017, search.svc.codfw.wmnet.
- 191 services: PDU infeed-load, Juniper alarms, OSPF status, VRRP status, Router interfaces, Juniper virtual chassis ports, gdnsd checkconf #page, authdns-update check.

Active checks enabled, but flagged as modified (notifications disabled by Puppet)
- MariaDB checks on db2202, db2230, db2901, db2902, db2903 (test-s1/test-s4) and db2233 (m2, including MariaDB read only m2 #page).
- haproxy failover on dbproxy2005, dbproxy2006, dbproxy2008.

Direction unknown
- Mask 2: pfw1-codfw (host and OSPF status); ferm check on aqs2006, wdqs2010, wikikube-worker2033, wikikube-worker2067, wikikube-worker2087, wikikube-worker2113, wikikube-worker2124, wikikube-worker2128, wikikube-worker2142.
- Mask 3: ps1-c4-codfw (host); infeed-load services on ps1-c4-codfw, ps1-c6-codfw, ps1-e2-codfw, ps1-e5-codfw, ps1-f4-codfw; SSH on an-worker1194, an-worker1197, db1245, druid-internal1004, krb1002; Backup freshness on backup1014; dump of s4 in codfw on backupmon1001; OpenSearch health check for shards on 9200 on cirrussearch2061; Auth DNS #page on ns1-v4, ns1-v6.

Command used to list them:

sudo awk -F= '
  /^(host|service)status \{/ { type=$0; sub(/status.*/, "", type); h=s=m=n=a="" }
  /^\thost_name=/              { h=$2 }
  /^\tservice_description=/    { s=substr($0, index($0,"=")+1) }
  /^\tmodified_attributes=/    { m=$2 }
  /^\tnotifications_enabled=/  { n=$2 }
  /^\tactive_checks_enabled=/  { a=$2 }
  /^\t\}/ && m!="" && m!=0     { print type "\t" h "\t" (type=="service" ? s : "-") "\t" m "\tnotif=" n "\tactive=" a }
' /path/to/status.dat

To check the Puppet defaults, compare with objects.cache (object_cache_file in icinga.cfg).

Full list

Hosts

Services

Related Objects
Search...

Status	Assigned	Task
In Progress	None	T439010 codfw power outage
Open	None	T439105 Cleanup manually modified attributes on Icinga hosts and services
Resolved	tappof	T439231 Check notification status for db1245
Resolved	tappof	T439232 Check notification status for druid-internal1004
Open	tappof	T439233 Check notification status for ps1-d5-eqiad

Event Timeline

Volans created this task.Thu, Sep 24, 8:27 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Sep 24, 8:27 AM

Aklapper edited projects, added: Sustainability (Incident Followup); removed: Wikimedia-Incident.Thu, Sep 24, 9:58 AM

From what I’ve seen in Puppet, active_checks_enabled is always enabled, except for passive checks, and there’s no way to manually override it. So I reset the modified attributes for the following services:

CHANGE_SVC_MODATTR;aqs2006;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;db2202;MariaDB Event Scheduler test-s1;0
CHANGE_SVC_MODATTR;db2202;MariaDB Replica IO: test-s1;0
CHANGE_SVC_MODATTR;db2202;MariaDB Replica Lag: test-s1;0
CHANGE_SVC_MODATTR;db2202;MariaDB Replica SQL: test-s1;0
CHANGE_SVC_MODATTR;db2202;mysqld processes;0
CHANGE_SVC_MODATTR;db2230;MariaDB Event Scheduler test-s4;0
CHANGE_SVC_MODATTR;db2230;MariaDB Events test-s4;0
CHANGE_SVC_MODATTR;db2230;MariaDB Replica IO: test-s4;0
CHANGE_SVC_MODATTR;db2230;MariaDB Replica Lag: test-s4;0
CHANGE_SVC_MODATTR;db2230;MariaDB Replica SQL: test-s4;0
CHANGE_SVC_MODATTR;db2230;mysqld processes;0
CHANGE_SVC_MODATTR;db2230;pt-heartbeat-wikimedia process;0
CHANGE_SVC_MODATTR;db2233;MariaDB Replica IO: m2;0
CHANGE_SVC_MODATTR;db2233;MariaDB Replica Lag: m2;0
CHANGE_SVC_MODATTR;db2233;MariaDB Replica SQL: m2;0
CHANGE_SVC_MODATTR;db2233;MariaDB read only m2 #page;0
CHANGE_SVC_MODATTR;db2233;mysqld processes;0
CHANGE_SVC_MODATTR;db2901;MariaDB Event Scheduler test-s4;0
CHANGE_SVC_MODATTR;db2901;MariaDB Events test-s4;0
CHANGE_SVC_MODATTR;db2901;MariaDB Replica IO: test-s4;0
CHANGE_SVC_MODATTR;db2901;MariaDB Replica Lag: test-s4;0
CHANGE_SVC_MODATTR;db2901;MariaDB Replica SQL: test-s4;0
CHANGE_SVC_MODATTR;db2901;mysqld processes;0
CHANGE_SVC_MODATTR;db2902;MariaDB Event Scheduler test-s4;0
CHANGE_SVC_MODATTR;db2902;MariaDB Events test-s4;0
CHANGE_SVC_MODATTR;db2902;MariaDB Replica IO: test-s4;0
CHANGE_SVC_MODATTR;db2902;MariaDB Replica Lag: test-s4;0
CHANGE_SVC_MODATTR;db2902;MariaDB Replica SQL: test-s4;0
CHANGE_SVC_MODATTR;db2902;mysqld processes;0
CHANGE_SVC_MODATTR;db2903;MariaDB Event Scheduler test-s4;0
CHANGE_SVC_MODATTR;db2903;MariaDB Events test-s4;0
CHANGE_SVC_MODATTR;db2903;MariaDB Replica IO: test-s4;0
CHANGE_SVC_MODATTR;db2903;MariaDB Replica Lag: test-s4;0
CHANGE_SVC_MODATTR;db2903;MariaDB Replica SQL: test-s4;0
CHANGE_SVC_MODATTR;db2903;mysqld processes;0
CHANGE_SVC_MODATTR;dbproxy2005;haproxy failover;0
CHANGE_SVC_MODATTR;dbproxy2006;haproxy failover;0
CHANGE_SVC_MODATTR;dbproxy2008;haproxy failover;0
CHANGE_SVC_MODATTR;pfw1-codfw;OSPF status;0
CHANGE_SVC_MODATTR;wdqs2010;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2033;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2067;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2087;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2113;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2124;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2128;Check whether ferm is active by checking the default input chain;0
CHANGE_SVC_MODATTR;wikikube-worker2142;Check whether ferm is active by checking the default input chain;0

Everything in Icinga with modified_attributes set matches the Puppet configuration, except:

┌─────────┬────────────────────┬──────────────────────────────────────────┬─────────────────────┬─────────┬──────────┬───────────────────┐
│  type   │        host        │                   svc                    │ modified_attributes │  notif  │  active  │  notif_on_puppet  │
│ varchar │      varchar       │                 varchar                  │        int32        │ varchar │ varchar  │      varchar      │
├─────────┼────────────────────┼──────────────────────────────────────────┼─────────────────────┼─────────┼──────────┼───────────────────┤
│ service │ db1245             │ SSH                                      │                   3 │ notif=1 │ active=1 │ notif_on_puppet=0 │
│ service │ druid-internal1004 │ SSH                                      │                   3 │ notif=1 │ active=1 │ notif_on_puppet=0 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-A-phase-X │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-A-phase-Y │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-A-phase-Z │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-B-phase-X │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-B-phase-Y │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ service │ ps1-d5-eqiad       │ ps1-d5-eqiad-infeed-load-tower-B-phase-Z │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
│ host    │ ps1-d5-eqiad       │ -                                        │                   1 │ notif=0 │ active=1 │ notif_on_puppet=1 │
└─────────┴────────────────────┴──────────────────────────────────────────┴─────────────────────┴─────────┴──────────┴───────────────────┘

I'm going to reset modified_attributes for everything else.

sudo awk -F= '
  /^(host|service)status \{/ { type=$0; sub(/status.*/, "", type); h=s=m=n=a="" }
  /^\thost_name=/              { h=$2 }
  /^\tservice_description=/    { s=substr($0, index($0,"=")+1) }
  /^\tmodified_attributes=/    { m=$2 }
  /^\tnotifications_enabled=/  { n=$2 }
  /^\tactive_checks_enabled=/  { a=$2 }
  /^\t\}/ && m!="" && m!=0     { print "\"" type "\";\"" h "\";\"" (type=="service" ? s : "-") "\";\"" m "\";\"notif=" n "\";\"active=" a "\""}
' /var/icinga-tmpfs/status.dat > /tmp/sh_i.csv

curl -G -H "Accept: application/json" http://localhost:8080/pdb/query/v4/resources   --data-urlencode 'query=["and", ["=","type", "Nagios_host"],["=","exported",true]]' | jq -r '.[] | (.certname | split(".")[0]) as $host | .parameters | select (.ensure == "present") | [ "host", $host , "-", "notif_on_puppet=\(.notifications_enabled)"] | @csv' | tr -s ',' ';'   > /tmp/h_p.csv

curl -G -H "Accept: application/json" http://localhost:8080/pdb/query/v4/resources   --data-urlencode 'query=["and", ["=","type", "Nagios_service"],["=","exported",true]]' | jq -r '.[] | .parameters | select (.ensure == "present") | [ "service", .host_name, .service_description, "notif_on_puppet=\(.notifications_enabled)"] | @csv' | tr -s ',' ';' > /tmp/s_p.csv

duckdb query:

.maxrows -1
select * from read_csv(
    'sh_i.csv',
    delim=';',
    strict_mode=false,
    header=false,
    columns={
        'type': 'VARCHAR',
        'host': 'VARCHAR',
        'svc': 'VARCHAR',
        'modified_attributes': 'INTEGER',
        'notif': 'VARCHAR',
        'active': 'VARCHAR'
    }
) icinga
join (
    select * from read_csv(
        's_p.csv',
        delim=';',
        strict_mode=false,
        header=false,
        columns={
            'type': 'VARCHAR',
            'host': 'VARCHAR',
            'svc': 'VARCHAR',
            'notif_on_puppet': 'VARCHAR',
        }
    )
    union all
    select * from read_csv(
        'h_p.csv',
        delim=';',
        strict_mode=false,
        header=false,
        columns={
            'type': 'VARCHAR',
            'host': 'VARCHAR',
            'svc': 'VARCHAR',
            'notif_on_puppet': 'VARCHAR',
        }
    )
) puppet using (type, host, svc)
WHERE split(icinga.notif, '=')[2]::INTEGER
   != split(puppet.notif_on_puppet, '=')[2]::INTEGER

Mentioned in SAL (#wikimedia-operations) [2026-09-25T09:43:35Z] <tappof> reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - T439105

tappof created subtask T439231: Check notification status for db1245.Fri, Sep 25, 10:28 AM

tappof created subtask T439232: Check notification status for druid-internal1004.

tappof created subtask T439233: Check notification status for ps1-d5-eqiad.Fri, Sep 25, 10:32 AM

Volans awarded a token.Fri, Sep 25, 10:41 AM

Marostegui closed subtask T439231: Check notification status for db1245 as Resolved.Fri, Sep 25, 1:06 PM

BTullis closed subtask T439232: Check notification status for druid-internal1004 as Resolved.Fri, Sep 25, 3:26 PM

Is there additional cleanup needed on this?