update failed puppet checks so that they go critical 24 hours
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jbond
	Oct 25 2019, 12:53 PM

Description

The icinga puppet test were recently updated so that they would only go into a warning state on a puppet failure and we instead only send a critical alert of a percentage of all hosts go into a failed state. however this has caused us to miss failing hosts which do a specific role and therefore don't trigger the percentage required. As such it would be useful to also go into an alerting state if puppet hasn't run for an extended peroiod of time. e.g. 24 hours

Details

	Subject	Repo	Branch	Lines +/-
	check_puppetrun: don't alert for disabled puppet agents for 1 day	operations/puppet	production	+8 -14

Customize query in gerrit

Related Objects

Mentioned Here: T236345: Icinga last puppet run check: re-enable relaxed per-host check

Event Timeline

jbond triaged this task as Medium priority.Oct 25 2019, 12:53 PM

jbond created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2019, 12:53 PM

Change 546165 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] check_puppetrun: dont alert for diabled puppet agents for 1 day

https://gerrit.wikimedia.org/r/546165

gerritbot added a project: Patch-For-Review.Oct 25 2019, 12:54 PM

I toko a look at how the mod gets the last puppet run data and iut just dose the following stat -c %Z /var/lib/puppet/state/classes.txt which isn't really usefull for our use case. I dont see any other meta data which gives the last failed state so we may need to store it our self

@jbond I had already opened T236345 for this. I guess that can probably be merged into this at this point.

@Volans merged, sorry i thought i had the action for the ticket :)

fgiunchedi moved this task from Inbox to In progress on the observability board.Oct 28 2019, 2:17 PM

herron renamed this task from update failed puppet checkes so that they go critical 24 hours to update failed puppet checks so that they go critical 24 hours.Oct 28 2019, 7:02 PM

herron updated the task description. (Show Details)

herron subscribed.

Change 546165 merged by Jbond:
[operations/puppet@production] check_puppetrun: don't alert for disabled puppet agents for 1 day

https://gerrit.wikimedia.org/r/546165

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2019, 2:10 PM

jbond added a project: User-jbond.Oct 30 2019, 5:24 PM

jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.Oct 30 2019, 5:33 PM

jbond moved this task from Active 🚁 to Patch for Review on the User-jbond board.Oct 31 2019, 9:58 AM

This is complete, please reopen if further issues/improvements

update failed puppet checks so that they go critical 24 hoursClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

update failed puppet checks so that they go critical 24 hours
Closed, ResolvedPublic
Actions