Page MenuHomePhabricator
Create Task
Maniphest T182759

Add Prometheus exporter to Jenkins instances
Open, LowPublic
Actions

Description

The CI Jenkins instances have the https://plugins.jenkins.io/monitoring monitoring plugin installed. It can exposes an endpoint for Prometheus to scape.

We might use the generic JMX exporter, though the Jenkins plugin might exposes a little more informations that are specific to Jenkins. The plugin is based on JavaMelody

Documentation: https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus

Jenkins instances I am aware of:

contint1001CI master
contint2001CI spare, stopped
releases1001for releasing
relaases1002spare

Related Objects
Search...

Event Timeline

hashar triaged this task as High priority.Dec 13 2017, 10:13 AM
hashar created this task.
hashar mentioned this in T177197: Export Prometheus-compatible JVM metrics from JVMs in production.
hashar updated the task description. (Show Details)Dec 13 2017, 10:16 AM
elukey removed a project: User-Elukey.Dec 18 2017, 9:20 AM
elukey moved this task from Backlog to Keep an eye on it on the User-Elukey board.
elukey added a project: User-Elukey.
fgiunchedi added a comment.Jan 3 2018, 11:59 AM

@hashar on what url are the metrics available? I tried localhost:8080/monitoring on contint1001 but yields 404:

contint1001:~$ curl localhost:8080/monitoring 
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /monitoring. Reason:
<pre>    Not Found</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>

</body>
</html>
fgiunchedi added a comment.Jan 3 2018, 12:00 PM

re: jmx-exporter we can deploy that in addition to jenkins' one, so we get standard/uniform jvm metrics as per parent task

• demon mentioned this in T184086: Add prometheus exporter to Gerrit.Jan 3 2018, 5:41 PM
hashar mentioned this in P6519 Jenkins monitoring plugin metrics for prometheus.Jan 3 2018, 6:01 PM
hashar added a comment.Jan 3 2018, 6:05 PM

The Jenkins instance uses /ci path prefix, and from https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#exposing-metrics-to-prometheus requires ?format=prometheus. The URLs would be:

A copy of all exposed metrics: P6519

We would need the Administer permission though:

contint1001:~$ curl http://localhost:8080/ci/monitoring?format=prometheus
$ curl 'https://integration.wikimedia.org/ci/monitoring?format=prometheus'
<html><head><meta http-equiv='refresh' content='1;url=/ci/login?from=%2Fci%2Fmonitoring%3Fformat%3Dprometheus'/><script>window.location.replace('/ci/login?from=%2Fci%2Fmonitoring%3Fformat%3Dprometheus');</script></head><body style='background-color:white; color:white;'>


Authentication required
<!--
You are authenticated as: anonymous
Groups that you are in:
  
Permission you need to have (but didn't): hudson.model.Hudson.Administer
-->

</body></html>

From the doc https://github.com/javamelody/javamelody/wiki/UserGuideAdvanced#5-security-with-a-collect-server we would want to tweak some settings:

-Djavamelody.plugin-authentication-disabled=true
-Djavamelody.allowed-addr-pattern=127.0.0.1  # IP of collecting server

Maybe we can just use the jmx-exporter , that most probably yields more or less the same metrics and would be good enough to monitor the java vm.

hashar added projects: Continuous-Integration-Infrastructure, Release-Engineering-Team.Jan 3 2018, 6:06 PM
greg moved this task from INBOX to Watching / External on the Release-Engineering-Team board.Feb 26 2018, 5:30 PM
greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.
dduvall claimed this task.Oct 1 2018, 8:54 PM
dduvall added a parent task: T205927: Collect and expose Jenkins build metrics for visualization, reporting, and analysis.
dduvall moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team (Watching / External). · View Herald TranscriptOct 1 2018, 8:54 PM
dduvall added a subscriber: thcipriani.Oct 1 2018, 9:07 PM

As of this morning both Jenkins master have the Prometheus plugin installed and enabled. The plugin will allows them to be used as Prometheus targets (at {jenkins-url}/prometheus) for collecting all sorts of build, node, and Jenkins master related metrics.

However, the plugin seems to have issues when "Fetch the test results of builds" is checked in the plugin configuration. DO NOT ENABLE THIS CONFIGURATION. @thcipriani and I observed high memory usage and request timeouts when this option was selected; we eventually tried killing the request thread and even then it continued to process for over 15 minutes.

We may have to go without individual metrics for tests and test suites for now, but the plugin as it's current configured provides a good starting point for Prometheus based metrics collection.

dduvall mentioned this in T205930: Set up a Prometheus server under the integration project.Oct 1 2018, 9:14 PM
fgiunchedi added a comment.Oct 2 2018, 8:25 AM
In T182759#4631716, @dduvall wrote:

As of this morning both Jenkins master have the Prometheus plugin installed and enabled. The plugin will allows them to be used as Prometheus targets (at {jenkins-url}/prometheus) for collecting all sorts of build, node, and Jenkins master related metrics.

However, the plugin seems to have issues when "Fetch the test results of builds" is checked in the plugin configuration. DO NOT ENABLE THIS CONFIGURATION. @thcipriani and I observed high memory usage and request timeouts when this option was selected; we eventually tried killing the request thread and even then it continued to process for over 15 minutes.

We may have to go without individual metrics for tests and test suites for now, but the plugin as it's current configured provides a good starting point for Prometheus based metrics collection.

Thanks for working on this! Happy to help with more Prometheus questions if needed. re: individual builds metrics as you noted exporting all builds as individual metrics is not feasible. Unless builds metrics get aggregated inside jenkins as a single/few metrics it sounds more like a logging use case than a metrics one (i.e. log build times to logstash)

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:38 PM
elukey removed a project: User-Elukey.Apr 16 2019, 11:00 AM
greg edited projects, added Release-Engineering-Team (Backlog); removed Release-Engineering-Team (Kanban).May 30 2019, 7:12 PM
• Phabricator_maintenance edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team (Backlog).Jun 12 2019, 11:51 PM
• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Later / Need volunteer on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:55 PM
greg added a project: Release-Engineering-Team.Jun 21 2019, 10:35 PM
greg edited projects, added Release-Engineering-Team (CI & Testing services); removed Release-Engineering-Team.Jun 24 2019, 7:51 PM
greg moved this task from Later / Need volunteer to Soon-ish on the Release-Engineering-Team-TODO board.Jul 6 2019, 4:55 AM
fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.Aug 27 2019, 1:46 PM
dduvall changed the task status from Open to Stalled.Oct 2 2019, 5:20 PM

Work on this has stalled. I've uninstalled the Prometheus plugin from Jenkins for now.

dduvall closed this task as Declined.Oct 2 2019, 5:21 PM

Marking this as "declined" to remove the task from view. We can always revive or reference this task should be pick the work back up.

hashar reopened this task as Open.Sep 17 2020, 6:49 AM
hashar added a project: observability.

Reopening, we need at least the JVM metrics to be exported so we can monitor its behavior. Part of T177197.

lmata moved this task from Inbox to Radar on the observability board.Oct 19 2020, 3:32 PM
Aklapper removed dduvall as the assignee of this task.Dec 19 2020, 11:29 AM
Aklapper added a subscriber: dduvall.

Removing task assignee due to inactivity, as this open task has been assigned to the same person for more than two years (see the emails sent to the task assignee on Oct27 and Nov23). Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.
(See https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.)

thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM
thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:42 AM
thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 3:58 AM
thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).
thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM
fgiunchedi removed a project: User-fgiunchedi.May 12 2021, 3:29 PM
thcipriani lowered the priority of this task from High to Low.May 13 2021, 3:39 AM
hashar moved this task from In-progress to Next on the Continuous-Integration-Infrastructure board.May 27 2021, 10:06 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits