While creating Grafana dashboards for the planet hosts I noticed no apache exporter is running on those hosts.
Having apache metrics would be a nice addition and should not be too much work. Also the empty dashboards in https://grafana-rw.wikimedia.org/d/xcoGtTASz/planet?orgId=1 would be gone then.
Change 1009575 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] planet: add prometheus apache exporter to role
Change 1009575 merged by Dzahn:
[operations/puppet@production] planet: add prometheus apache exporter to role
ii prometheus-apache-exporter 0.12.0-1+b4
At least the package is installed now on planet* servers.
20:14 < denisse> Hello mutante, if the metrics are already available via HTTP, the next step would be to configure Prometheus to scrape these metrics by adding a new job.
20:14 < denisse> I've included a link to our documentation on this subject for your reference. If you have any further inquiries, please don't hesitate to reach out:
https://wikitech.wikimedia.org/wiki/Prometheus#Adding_new_metrics
Currently the exporter fails with
prometheus-apache-exporter[226585]: prometheus-apache-exporter: error: unknown short flag '-s', try --help
Probably related to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/prometheus/manifests/apache_exporter.pp#11. But I'm not sure if that has changed in the newer bookworm version of the exporter. Probably the argument has to be --scrape_uri in the newer version?
Change 1009775 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] apache_exporter: fix argument syntax in bookworm
@Jelto I confirmed the change is that on buster and bullseye the commandline args of the exporter start with "-" but in bookworm they now start with "--".
phab1004:~] $ prometheus-apache-exporter --help
Usage of prometheus-apache-exporter:
..
-scrape_uri string
URI to apache stub status page. (default "http://localhost/server-status/?auto")[planet1003:~] $ prometheus-apache-exporter --help
usage: prometheus-apache-exporter [<flags>]
Flags:
-h, --help Show context-sensitive help (also try --help-long and --help-man).
..
--scrape_uri="http://localhost/server-status/?auto"
URI to apache stub status page.Uploaded a patch to fix this.
Change #1009775 merged by Dzahn:
[operations/puppet@production] prometheus/apache_exporter: drop argument parameter
After https://gerrit.wikimedia.org/r/1009775 has been deployed and entirely dropped the argument parameter the package is now installed again on planet* hosts and the service doesn't fail to start any longer.
So it can now generally be used on bookworm servers.
root@mw1413:/# curl -s 10.64.32.132:9117/metrics | tail -n 1
process_virtual_memory_max_bytes -1
root@mw1413:/# lsb_release -c
Codename: buster
---
root@planet1003:/# curl -s 10.64.16.112:9117/metrics | tail -n1
promhttp_metric_handler_requests_total{code="503"} 0
root@planet1003:/# lsb_release -c
..
Codename: bookwormChange #1014116 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] prometheus/ops: add config for scraping apache metrics on planet servers
Change #1014116 merged by Dzahn:
[operations/puppet@production] prometheus/ops: add config for scraping apache metrics on planet servers
@Jelto After also adding a job to scrape the metrics I am now seeing _some_ data trickle in here:
https://grafana.wikimedia.org/d/xcoGtTASz/planet?orgId=1&var-node=planet1003
Like "Apache up".