Page MenuHomePhabricator

Add prometheus exporter to Gerrit
Open, NormalPublic

Description

Requires upgrade to 2.14.x first, but there's now metrics plugins available and one of them is JMX. Let's do it.

Upstream plugin

Cf: T182759 for Jenkins.

Related Objects

StatusAssignedTask
ResolvedTgr
ResolvedPaladox
Resolveddemon
ResolvedNone
ResolvedNone
OpenNone
OpenNone
ResolvedPaladox
ResolvedPaladox
ResolvedPaladox
OpenNone
Resolveddemon
ResolvedPaladox
DeclinedNone
ResolvedPaladox
ResolvedNone
OpenNone
Resolveddemon
ResolvedNone
ResolvedPaladox
ResolvedPaladox
Resolveddemon
OpenNone

Event Timeline

demon triaged this task as Normal priority.Jan 3 2018, 5:41 PM
demon created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2018, 5:41 PM
demon added a comment.Jan 3 2018, 5:42 PM

Right after posting, I notice there's a stable-2.13 branch, so maybe I don't need to wait for the upgrade?

To give an outline of what's needed:

  • install prometheus-jmx-exporter debian package
  • tweak jvm's options to include -javaagent ...
  • ship an empty configuration file for jmx-exporter
  • check resulting metrics and tweak the configuration as needed

In puppet we've already got examples of other services doing the same, e.g. analytics daemons (by @elukey)

Paladox added a subscriber: Paladox.Jan 3 2018, 5:51 PM

@demon do you want to add this plugin in your 2.14 patch or will you do a follow up adding this?

demon added a comment.Jan 29 2018, 3:24 AM

We'll do it separately.

We can do this now :)

demon moved this task from Bugs & stuff to Local hacks on the Gerrit board.Feb 13 2018, 4:02 AM

Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/

The javamelody plugin seems much more active: https://gerrit.googlesource.com/plugins/javamelody (got an update just today)

it also seems to support exposing metrics to Prometheus: https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus

Thanks @thcipriani for working on this! From a quick look it seems javamelody offers a deeper look into the jvm rather than pure JMX and exposes a read/write interface (e.g. you can force a gc run). re: prometheus metrics those are also protected by username/password AFAICS, meaning to scrape them we'll need to put credentials in prometheus config and sent those along in the http request? I've put a capture of said metrics in https://phabricator.wikimedia.org/P7810 for reference.

At any rate, I still think prometheus-jmx-exporter makes sense for gerrit's JVM too (in addition to javamelody? seems possible), if anything to be consistent with instrumentation from all other JVMs (i.e. T177197). What do you think?

Paladox added a comment.EditedJan 31 2019, 3:50 PM

This would have been useful for debugging T215004 , should we triage as high?

crusnov added a subscriber: crusnov.May 6 2019, 9:02 PM

Just to +1 the idea of shipping javamelody to prometheus. Let me know if I can help at all.

@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.

@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.

Okay the general approach will be to have a dummy token in labsprivate hiera repository (see https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet) whereas I can put a token in the real private. Then this token will be referred to in the gerrit configuration and in the prometheus configuration which will end up something similar to the scraping configuration in modules/profile/manifests/prometheus/k8s.pp (thanks @fgiunchedi :).

So the starting place is to get the gerrit puppet change and the labsprivate change in, and we can move from there.

Change 508621 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set plugin.javamelody.prometheusBearerToken

https://gerrit.wikimedia.org/r/508621

Change 508952 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Add prometheus server for gerrit javamelody monitoring

https://gerrit.wikimedia.org/r/508952

Change 508621 merged by CRusnov:
[operations/puppet@production] Gerrit: Set plugin.javamelody.prometheusBearerToken

https://gerrit.wikimedia.org/r/508621

Change 508952 merged by CRusnov:
[operations/puppet@production] Add prometheus server for gerrit javamelody monitoring

https://gerrit.wikimedia.org/r/508952

hashar added a subscriber: hashar.Jul 17 2019, 12:45 PM

Status

There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.org/r/monitoring?format=prometheus . It reports JVM related metrics which are rendered on Grafana at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody . That solely report Java internal metrics.


Gerrit also has its own metrics as listed on https://gerrit.wikimedia.org/r/Documentation/metrics.html and they can be exposed via plugins. It seems we would to investigate either of:

metrics-reporter-jmx

The code is a single java file with just a few lines of code. It just adds all the Gerrit metrics to the Jmx reporter. The devil question is: is that used by the already installed javamelody plugin there are apparently some hints on https://github.com/javamelody/javamelody/issues/767

https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/

metrics-reporter-prometheus

It seems to add a brand new prometheus entry point with bearer token and a View Metrics global capability. Not sure we want all that additional complexity when we already have a Prometheus exporter. Maybe it is in the end easier to just add a second Prometheus endpoint dedicated to the Gerrit metrics.

https://gerrit.googlesource.com/plugins/metrics-reporter-prometheus/

In the end, I guess we need to try them out? :-]

I have quickly talked with @Paladox about it. He has tried the metrics-reporter-prometheus plugin and it does exposes all metrics to another endpoint. So we should add that plugin to our deployment, configure it, add a prometheus server and potentially we would have bunch of new metrics to build a dashboard with.

The endpoint might end up being https://gerrit.wikimedia.org/r/plugins/metrics-reporter-prometheus/metrics based on the documentation. That requires an authenticated request and the user to be assigned the View Metrics capability.

metrics-reporter-jmx
The code is a single java file with just a few lines of code. It just adds all the Gerrit metrics to the Jmx reporter. The devil question is: is that used by the already installed javamelody plugin there are apparently some hints on https://github.com/javamelody/javamelody/issues/767
https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/

I'd be in favor of something like this.

Sort of orthogonal, please consider also adding the jmx exporter (even if it doesn't have gerrit-specific metrics yet) to get jvm general metrics (context is T177197: Export Prometheus-compatible JVM metrics from JVMs in production)

Sort of orthogonal, please consider also adding the jmx exporter (even if it doesn't have gerrit-specific metrics yet) to get jvm general metrics (context is T177197: Export Prometheus-compatible JVM metrics from JVMs in production)

The jmx export for the JVM has already been setup via this T184086 task, and from my summary:

There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.org/r/monitoring?format=prometheus . It reports JVM related metrics which are rendered on Grafana at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody . That solely report Java internal metrics.

The devil is to now have the Gerrit internal metrics to be reported backup :-] I guess we just need to setup a Gerrit with each plugins and see what happens / what has to be configured.

Sort of orthogonal, please consider also adding the jmx exporter (even if it doesn't have gerrit-specific metrics yet) to get jvm general metrics (context is T177197: Export Prometheus-compatible JVM metrics from JVMs in production)

The jmx export for the JVM has already been setup via this T184086 task, and from my summary:

There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.org/r/monitoring?format=prometheus . It reports JVM related metrics which are rendered on Grafana at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody . That solely report Java internal metrics.

The devil is to now have the Gerrit internal metrics to be reported backup :-] I guess we just need to setup a Gerrit with each plugins and see what happens / what has to be configured.

I think we're talking about two different things, what I'm referring to is jmx_exporter like we are deploying to the rest of JVMs, essentially the procedure I've outlined here https://phabricator.wikimedia.org/T184086#3872132, it doesn't look like that has been added yet? I believe it should, in addition to the existing exporters/plugins for gerrit that is.