Page MenuHomePhabricator

Puppet: enable reports to puppetdb
Closed, ResolvedPublic

Description

We should investigate and carefully enable the Puppet reports to puppetdb too, in addition to servermon for the moment, in order to slowly offload servermon from its functionalities.

Given that we now have Puppetboard as a PuppetDB web UI, it will also nicely show all the report-related information.

Event Timeline

Volans triaged this task as Medium priority.Mar 28 2018, 10:58 AM
Volans created this task.

Mentioned in SAL (#wikimedia-operations) [2018-03-28T10:59:43Z] <volans> performing a few minutes live test of reporting Puppet reports to puppetdb too on puppetmaster1001 - T190918

Change 422907 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Puppetmaster: store reports also in puppetdb

https://gerrit.wikimedia.org/r/422907

From the quick test I've made yesterday enabling reporting also to puppetdb for some minutes, I got ~200 hosts reported and showing data in Puppetboard, I didn't notice any sensible load/ram/disk usage increase on puppetdb hosts.
Moreover our report-ttl parameter is set to 1d, so I don't expect this huge amount of data to be kept longer term.

Of course though this should be enabled with care across the puppetmasters and monitored for the first days to be sure we don't have pileups or contention when puppetdb cleanup the old data.

The plan as of now is to enable it on next Tuesday, to avoid issues in the long weekend.

Mentioned in SAL (#wikimedia-operations) [2018-04-03T08:40:57Z] <volans> temporarily disabled puppet (and re-enabling it one-by-one) on all prod puppetmasters to deploy g/422907 - T190918

Change 422907 merged by Volans:
[operations/puppet@production] Puppetmaster: store reports also in puppetdb

https://gerrit.wikimedia.org/r/422907

Volans claimed this task.

Reports are enabled since ~1 day without any incident. Resolving.