To give an outline of what's needed:
- install prometheus-jmx-exporter debian package
- tweak jvm's options to include -javaagent ...
- ship an empty configuration file for jmx-exporter
- check resulting metrics and tweak the configuration as needed
In puppet we've already got examples of other services doing the same, e.g. analytics daemons (by @elukey)
Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/
The javamelody plugin seems much more active: https://gerrit.googlesource.com/plugins/javamelody (got an update just today)
it also seems to support exposing metrics to Prometheus: https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus
Thanks @thcipriani for working on this! From a quick look it seems javamelody offers a deeper look into the jvm rather than pure JMX and exposes a read/write interface (e.g. you can force a gc run). re: prometheus metrics those are also protected by username/password AFAICS, meaning to scrape them we'll need to put credentials in prometheus config and sent those along in the http request? I've put a capture of said metrics in https://phabricator.wikimedia.org/P7810 for reference.
At any rate, I still think prometheus-jmx-exporter makes sense for gerrit's JVM too (in addition to javamelody? seems possible), if anything to be consistent with instrumentation from all other JVMs (i.e. T177197). What do you think?
@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.
Okay the general approach will be to have a dummy token in labsprivate hiera repository (see https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet) whereas I can put a token in the real private. Then this token will be referred to in the gerrit configuration and in the prometheus configuration which will end up something similar to the scraping configuration in modules/profile/manifests/prometheus/k8s.pp (thanks @fgiunchedi :).
So the starting place is to get the gerrit puppet change and the labsprivate change in, and we can move from there.