To give an outline of what's needed:
- install prometheus-jmx-exporter debian package
- tweak jvm's options to include -javaagent ...
- ship an empty configuration file for jmx-exporter
- check resulting metrics and tweak the configuration as needed
In puppet we've already got examples of other services doing the same, e.g. analytics daemons (by @elukey)
Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/
The javamelody plugin seems much more active: https://gerrit.googlesource.com/plugins/javamelody (got an update just today)
it also seems to support exposing metrics to Prometheus: https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus
Thanks @thcipriani for working on this! From a quick look it seems javamelody offers a deeper look into the jvm rather than pure JMX and exposes a read/write interface (e.g. you can force a gc run). re: prometheus metrics those are also protected by username/password AFAICS, meaning to scrape them we'll need to put credentials in prometheus config and sent those along in the http request? I've put a capture of said metrics in https://phabricator.wikimedia.org/P7810 for reference.
At any rate, I still think prometheus-jmx-exporter makes sense for gerrit's JVM too (in addition to javamelody? seems possible), if anything to be consistent with instrumentation from all other JVMs (i.e. T177197). What do you think?
@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.
Okay the general approach will be to have a dummy token in labsprivate hiera repository (see https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet) whereas I can put a token in the real private. Then this token will be referred to in the gerrit configuration and in the prometheus configuration which will end up something similar to the scraping configuration in modules/profile/manifests/prometheus/k8s.pp (thanks @fgiunchedi :).
So the starting place is to get the gerrit puppet change and the labsprivate change in, and we can move from there.
There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.org/r/monitoring?format=prometheus . It reports JVM related metrics which are rendered on Grafana at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody . That solely report Java internal metrics.
Gerrit also has its own metrics as listed on https://gerrit.wikimedia.org/r/Documentation/metrics.html and they can be exposed via plugins. It seems we would to investigate either of:
The code is a single java file with just a few lines of code. It just adds all the Gerrit metrics to the Jmx reporter. The devil question is: is that used by the already installed javamelody plugin there are apparently some hints on https://github.com/javamelody/javamelody/issues/767
It seems to add a brand new prometheus entry point with bearer token and a View Metrics global capability. Not sure we want all that additional complexity when we already have a Prometheus exporter. Maybe it is in the end easier to just add a second Prometheus endpoint dedicated to the Gerrit metrics.
In the end, I guess we need to try them out? :-]
I have quickly talked with @Paladox about it. He has tried the metrics-reporter-prometheus plugin and it does exposes all metrics to another endpoint. So we should add that plugin to our deployment, configure it, add a prometheus server and potentially we would have bunch of new metrics to build a dashboard with.
The endpoint might end up being https://gerrit.wikimedia.org/r/plugins/metrics-reporter-prometheus/metrics based on the documentation. That requires an authenticated request and the user to be assigned the View Metrics capability.
I'd be in favor of something like this.
Sort of orthogonal, please consider also adding the jmx exporter (even if it doesn't have gerrit-specific metrics yet) to get jvm general metrics (context is T177197: Export Prometheus-compatible JVM metrics from JVMs in production)
I think we're talking about two different things, what I'm referring to is jmx_exporter like we are deploying to the rest of JVMs, essentially the procedure I've outlined here https://phabricator.wikimedia.org/T184086#3872132, it doesn't look like that has been added yet? I believe it should, in addition to the existing exporters/plugins for gerrit that is.