Page MenuHomePhabricator

Add prometheus exporter to Gerrit
Open, NormalPublic

Description

Requires upgrade to 2.14.x first, but there's now metrics plugins available and one of them is JMX. Let's do it.

Upstream plugin

Cf: T182759 for Jenkins.

Related Objects

StatusAssignedTask
ResolvedTgr
ResolvedPaladox
Resolveddemon
ResolvedNone
ResolvedNone
OpenNone
OpenNone
ResolvedPaladox
ResolvedPaladox
ResolvedPaladox
OpenNone
Resolveddemon
ResolvedPaladox
DeclinedNone
ResolvedPaladox
ResolvedNone
OpenNone
Resolveddemon
ResolvedNone
ResolvedPaladox
ResolvedPaladox
Resolveddemon
OpenNone

Event Timeline

demon created this task.Jan 3 2018, 5:41 PM
demon triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2018, 5:41 PM
demon added a comment.Jan 3 2018, 5:42 PM

Right after posting, I notice there's a stable-2.13 branch, so maybe I don't need to wait for the upgrade?

To give an outline of what's needed:

  • install prometheus-jmx-exporter debian package
  • tweak jvm's options to include -javaagent ...
  • ship an empty configuration file for jmx-exporter
  • check resulting metrics and tweak the configuration as needed

In puppet we've already got examples of other services doing the same, e.g. analytics daemons (by @elukey)

Paladox added a subscriber: Paladox.Jan 3 2018, 5:51 PM

@demon do you want to add this plugin in your 2.14 patch or will you do a follow up adding this?

demon added a comment.Jan 29 2018, 3:24 AM

We'll do it separately.

We can do this now :)

demon moved this task from Bugs & stuff to Local hacks on the Gerrit board.Feb 13 2018, 4:02 AM

Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/

The javamelody plugin seems much more active: https://gerrit.googlesource.com/plugins/javamelody (got an update just today)

it also seems to support exposing metrics to Prometheus: https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus

Thanks @thcipriani for working on this! From a quick look it seems javamelody offers a deeper look into the jvm rather than pure JMX and exposes a read/write interface (e.g. you can force a gc run). re: prometheus metrics those are also protected by username/password AFAICS, meaning to scrape them we'll need to put credentials in prometheus config and sent those along in the http request? I've put a capture of said metrics in https://phabricator.wikimedia.org/P7810 for reference.

At any rate, I still think prometheus-jmx-exporter makes sense for gerrit's JVM too (in addition to javamelody? seems possible), if anything to be consistent with instrumentation from all other JVMs (i.e. T177197). What do you think?

Paladox added a comment.EditedJan 31 2019, 3:50 PM

This would have been useful for debugging T215004 , should we triage as high?

crusnov added a subscriber: crusnov.Mon, May 6, 9:02 PM

Just to +1 the idea of shipping javamelody to prometheus. Let me know if I can help at all.

@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.

@crusnov we could use your help, yup. We need to create a prometheusBearerToken [plugin.javamelody.prometheusBearerToken] https://gerrit.googlesource.com/plugins/javamelody/+/refs/heads/stable-2.15/src/main/resources/Documentation/config.md .Which then get prometheus to query gerrit.wikimedia.org/r/monitoring?format=prometheus using the token.

Okay the general approach will be to have a dummy token in labsprivate hiera repository (see https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet) whereas I can put a token in the real private. Then this token will be referred to in the gerrit configuration and in the prometheus configuration which will end up something similar to the scraping configuration in modules/profile/manifests/prometheus/k8s.pp (thanks @fgiunchedi :).

So the starting place is to get the gerrit puppet change and the labsprivate change in, and we can move from there.

Change 508621 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set plugin.javamelody.prometheusBearerToken

https://gerrit.wikimedia.org/r/508621

Change 508952 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Add prometheus server for gerrit javamelody monitoring

https://gerrit.wikimedia.org/r/508952

Change 508621 merged by CRusnov:
[operations/puppet@production] Gerrit: Set plugin.javamelody.prometheusBearerToken

https://gerrit.wikimedia.org/r/508621

Change 508952 merged by CRusnov:
[operations/puppet@production] Add prometheus server for gerrit javamelody monitoring

https://gerrit.wikimedia.org/r/508952