Monitoring: create an alert for daemonized puppet
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Volans
	May 26 2017, 8:24 AM

Description

We've seen in T166203#3294061 that a simple mistake can leave puppet damonized on the hosts that keep running. To avoid this we should add an alert to monitor and ensure no puppet process is running since more than says 2 hours. The check can be done infrequently, like once per hour or so IMHO.

Having a daemonized puppet on the host conflict with our cron-based puppet runs and generates any sort of flapping issues that are hard to find.

Some example of running puppet were:

/usr/bin/ruby /usr/bin/puppet agent 0tv
/usr/bin/ruby /usr/bin/puppet agent .-t
/usr/bin/ruby /usr/bin/puppet agent -d

Details

	Subject	Repo	Branch	Lines +/-
	base/puppet: use "false" instead of "no" with "daemonize" option	operations/puppet	production	+1 -1
	base/puppet: add "daemonize = no" to agent config	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T166203: Upgrade facter to version 2.4.6
Mentioned Here: T166203: Upgrade facter to version 2.4.6

Event Timeline

Volans created this task.May 26 2017, 8:24 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2017, 8:24 AM

Volans mentioned this in T166203: Upgrade facter to version 2.4.6.May 26 2017, 8:24 AM

Volans updated the task description. (Show Details)

faidon assigned this task to Dzahn.May 26 2017, 8:35 AM

Change 358501 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/puppet: add "daemonize = no" to agent config

https://gerrit.wikimedia.org/r/358501

gerritbot added a project: Patch-For-Review.Jun 13 2017, 3:16 AM

Change 358501 merged by Faidon Liambotis:
[operations/puppet@production] base/puppet: add "daemonize = no" to agent config

https://gerrit.wikimedia.org/r/358501

The change above makes it impossible to have daemonized agents running as root so I 'd say this is resolved and we might be spared to need to have monitoring for this.

Volans moved this task from Backlog to Done on the SRE-tools board.Jun 13 2017, 8:27 AM

Yep, i was hoping for this to be the outcome. I'll call it resolved then. :)

Change 359084 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/puppet: use "false" instead of "no" with "daemonize" option

https://gerrit.wikimedia.org/r/359084

uhmm.. i think we have to use "false" instead of "no". Well, the confusion is that the command line parameters are with "no" but in the config file it expects a Boolean (depending on version?).

What happened is I saw that host dumpsdata1001 had a CRIT systemd state in Icinga.

I checked which unit was failed and it was puppet, and i saw "Invalid value '"no"' for boolean parameter: daemonize" .. ugh :/

BUT: I didn't see that on other hosts before and i did test that and it seemed to work and prevented it from daemonizing.

Also, the original "no" was advice directly from puppet people on Puppet in Freenode.
On https://docs.puppet.com/puppet/latest/configuration.html#daemonize there is:

When using boolean settings on the command line, use --setting and --no-setting instead of --setting (true|false). (Using --setting false results in “Error: Could not parse application options: needless argument”.)

so .. https://gerrit.wikimedia.org/r/#/c/359084/

wanna confirm?

Change 359084 merged by Faidon Liambotis:
[operations/puppet@production] base/puppet: use "false" instead of "no" with "daemonize" option