As discovered in https://phabricator.wikimedia.org/T273064#6805181 the icinga meta-monitoring needs to be updated as we're removing contacts from legacy pager system.
This is currently causing systemd daemon sync_check_icinga_contacts.service to error with Failed validation of new contacts file. Aborting. on alert1001
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| icinga: reduce contacts limit to 1 | operations/software/external-monitoring | master | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | Volans | T273951 Update Icinga meta-monitoring to account for "no pagers" in contacts |
Change 662973 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/external-monitoring@master] icinga: reduce contacts limit to 1
Change 662973 merged by jenkins-bot:
[operations/software/external-monitoring@master] icinga: reduce contacts limit to 1
Mentioned in SAL (#wikimedia-operations) [2021-02-09T14:26:28Z] <volans> cd /srv/external-monitoring; git fetch/status/pull on wikitech-static - T273951
Feb 9 14:26:47 alert1001 sync-check-icinga-contacts[15727]: VALID: /etc/check_icinga/config.yaml is valid according to /srv/external-monitoring/icinga/check_icinga_config.schema Feb 9 14:26:47 alert1001 sync-check-icinga-contacts[15727]: VALID: /etc/check_icinga/contacts.yaml is valid according to /srv/external-monitoring/icinga/check_icinga_contacts.schema Feb 9 14:26:47 alert1001 sync-check-icinga-contacts[15727]: Successfully synced new configuration
Leaving it open to check that alert2001 will self-fix itself the next time the timer triggers the unit and resolve afterwards.