Page MenuHomePhabricator

PuppetDB is auto-deactivating hosts
Closed, ResolvedPublic

Description

It happened that ms-be1012 was deactivated from PuppetDB automatically after it has been with puppet disabled for 14 days due to a disk issue.

This is because we use the node-ttl setting:
https://github.com/wikimedia/puppet/blob/ddc661fb5304b49bc8b45f0f5bdb8804d294fc91/modules/puppetmaster/manifests/puppetdb.pp#L66

That deactivate the host after 14d: https://docs.puppet.com/puppetdb/2.3/configure.html#node-ttl

While we all agree that hosts shouldn't be with puppet disabled for longer time, I think that in any case hosts shouldn't disappear either from PuppetDB given that in turn this will remove it from Icinga, known hosts file, making it a ghost with the risk to loose track of it.

Thoughts?

Event Timeline

I'm +1 on not deactivating hosts automatically in puppetdb. Disabling auto-deactivation would also let us automatically catch hosts that have been decommissioned in DNS but haven't been node-cleaned from puppetdb I think

Ottomata triaged this task as Medium priority.
Ottomata subscribed.

+1 from me too.

@Volans, I'm just triaging, feel free to assign un-assign this at will.

Yeah, let's disable it until we have a good reason for enabling it. It will probably need a restart of puppetdb which will be followed by a storm of puppet alerts btw

@akosiaris you are correct, but I think that's inevitable.

Change 346110 had a related patch set uploaded (by Volans):
[operations/puppet@production] Puppet: do not deactivate hosts in PuppetDB automatically

https://gerrit.wikimedia.org/r/346110

@Joe @akosiaris, actually looks like this is a NOOP on the puppetmasters, but a change on just the puppetdb hosts:

$ sudo cumin --dry-run 'R:class = puppetmaster::puppetdb' date
2 hosts will be targeted:
nihal.codfw.wmnet,nitrogen.eqiad.wmnet
DRY-RUN mode enabled, aborting

That seems confirmed by the puppet compiler results:
https://puppet-compiler.wmflabs.org/6003/
https://puppet-compiler.wmflabs.org/6004/

Ops, I read the previous message as it required a restart of puppetmasters, not puppetdb, sorry for the misunderstanding.

Change 346110 merged by Volans:
[operations/puppet@production] Puppet: do not deactivate hosts in PuppetDB automatically

https://gerrit.wikimedia.org/r/346110

Mentioned in SAL (#wikimedia-operations) [2017-04-05T11:42:33Z] <volans> disabling ircecho for the merge of gerrit/346110 ( T159163 ) and postgres upgrade