Page MenuHomePhabricator

Puppetdb: not refreshed on config change?
Closed, ResolvedPublic

Description

It seems that when we change Puppetdb's configuration files, Puppet doesn't refresh the service:

Computing checksum on file /etc/puppetdb/conf.d/database.ini
Filebucketed /etc/puppetdb/conf.d/database.ini to puppet with sum cf4b759d05f1e168c9ce4a66cdedf24b
content changed '{md5}cf4b759d05f1e168c9ce4a66cdedf24b' to '{md5}97c0bd8aeadebfb784269d99b7d0db3c'
Applied catalog in 21.70 seconds

From a cursory look at the documentation it says:

All other settings, which go in PuppetDB’s configuration file(s) and take effect after the service is restarted.

I'm not sure if the current behaviour is expected or not, because auto-refresh might cause puppet failures, but if the above is true we should ensure that the unit gets refreshed (automatically or manually via a cookbook or something) on file change.

This came up while looking at T291539

Event Timeline

Volans triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Currently the this is a conscious decisions. when puppetdb is restarting all submissions to it are rejected which generally cause the "wide spread puppet issues" alert. Recently i discovered you can send puppetdb a HUP signal and it will refresh its config (how much of its config is unknown as i don't see much documentation) about this. Sending a HUP signal means that the service reloads in ~1.5-2 seconds vs ~30-40+ seconds for a restart. so we could consider updating things to reload and accept that we made get the odd transient failure.

but if the above is true we should ensure that the unit gets refreshed (automatically or manually via a cookbook or something) on file change.

the general process is manual, we deploy the change. then restart/reload the pupetdb services, ensuring we disable puppet as per the service restarts page

It is worth mentioning that puppetdb is not alone in this behaviour we have many services which require a manual restart e.g. postgresql, mariadb, likely many more as such it would be nice to have a general solution which alerts if a damon needs restarting because a new config files has been deployed, not sure of the top of my head how to implement that, two big issues i see:

  1. how to we know which config file a process has loaded
  2. how to check the last time a service revived a HUP (restart is obviously easier)

Then we can make use of the reboot base cookbook to create cookbooks to preform the restarts

jbond claimed this task.

@Volans im going to reject this and say its better to manually disable puppet fleet wide to roll out theses changes but please re-open if you disagree