puppetdb prometheus metrics per-host metrics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Jul 18 2019, 10:06 AM

Description

While investigating something else I discovered that puppetdb hosts are exporting a ton of metrics, with the hostname in the metric name. Prometheus isn't happy about that and we are already blacklisting some mbeans, likely the blacklist needs to be expanded.

puppetdb1001:~$ curl -s 10.64.32.214:9400/metrics | grep ^puppetlabs_puppetdb_http_pdb_query_v4_catalogs | wc -l
57060
puppetdb1001:~$ curl -s 10.64.32.214:9400/metrics | grep ^puppetlabs_puppetdb_http_pdb_query_v4_catalogs | head -2
puppetlabs_puppetdb_http_pdb_query_v4_catalogs_mw2231_codfw_wmnet_resources_service_time_98thPercentile 480.154882
puppetlabs_puppetdb_http_pdb_query_v4_catalogs_wdqs1010_eqiad_wmnet_service_time_50thPercentile 300.70412999999996

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	puppetmaster: blacklist per-host catalogs metrics	operations/puppet	production	+1 -0

Customize query in gerrit

Event Timeline

fgiunchedi created this task.Jul 18 2019, 10:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2019, 10:06 AM

Change 524182 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] puppetmaster: blacklist per-host catalogs metrics

https://gerrit.wikimedia.org/r/524182

gerritbot added a project: Patch-For-Review.Jul 18 2019, 10:17 AM

below are all the hosts with puppetdb package installed

af-puppetdb[01-02].automation-framework.eqiad.wmflabs,compiler[1001-1002].puppet-diffs.eqiad.wmflabs 
jeh-puppetdb.testlabs.eqiad.wmflabs     
toolsbeta-puppetdb-01.toolsbeta.eqiad.wmflabs
deployment-puppetdb02.deployment-prep.eqiad.wmflabs
puppet-paladox.git.eqiad.wmflabs,puppet-phabricator.phabricator.eqiad.wmflabs

Change 524182 merged by Filippo Giunchedi:
[operations/puppet@production] puppetmaster: blacklist per-host catalogs metrics

https://gerrit.wikimedia.org/r/524182

Change deployed, the high cardinality metrics should also be deleted. To do that we'll need to pass --web.enable-admin-api to prometheus first to be able to delete metrics: https://www.robustperception.io/deleting-time-series-from-prometheus

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2019, 11:10 AM

herron triaged this task as Medium priority.Jul 23 2019, 2:03 PM

cc @EBernhardson this is likely one of the problems you have been experiencing with prometheus' web interface (i.e. dropdown/autocomplete is slow because of many metric names)

fgiunchedi added a project: User-fgiunchedi.Aug 19 2019, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T10:53:34Z] <godog> temporarily enable prometheus admin web api in prometheus@ops in eqiad to delete spammy metrics - T228395

prometheus1004 completed, with this process:

# depool
# stop puppet
# add  --web.enable-admin-api to /lib/systemd/system/prometheus@ops.service
systemctl daemon-reload
systemctl restart prometheus@ops
curl -v -X POST -g 'http://localhost:9900/ops/api/v1/admin/tsdb/delete_series?match[]={__name__=~"^puppetlabs_puppetdb_http_pdb_query_v4_catalogs_.*wmnet.*"}'
curl -v -X POST -g 'http://localhost:9900/ops/api/v1/admin/tsdb/delete_series?match[]={__name__=~"^puppetlabs_puppetdb_http_pdb_query_v4_catalogs_.*wikimedia.*"}'
curl -XPOST http://localhost:9900/ops/api/v1/admin/tsdb/clean_tombstones
# wait for tombstones to get cleaned, the curl will return once done
# reenable puppet agent, prometheus will get restarted
# verify all is well
$ curl -s http://localhost:9900/ops/api/v1/label/__name__/values | jq '.data | length'
7004
# repool

Both prometheus1003 and prometheus1004 have been cleaned and repooled, resolving. @EBernhardson please give the web ui another try, autocomplete should be significantly faster now

puppetdb prometheus metrics per-host metricsClosed, ResolvedPublicActions

Description

Details

Event Timeline

puppetdb prometheus metrics per-host metrics
Closed, ResolvedPublic
Actions