Page MenuHomePhabricator
Create Task
Maniphest T181728

Stop using jmx_exporter deployed via scap in favour of Debian package
Closed, ResolvedPublic
Actions

Description

jmx_exporter is packaged as Debian (by us) prometheus-jmx-exporter, we should switch current users of jmx_exporter deployed via scap (only cassandra-restbase afaik) to use the Debian package instead.

Details

SubjectRepoBranchLines +/-
cassandra: switch to using jmx-exporter jar from Debian packageoperations/puppetproduction+2 -2
cassandra: flip jmx exporter's jar referenceoperations/puppetproduction+3 -3
cassandra: use prometheus-jmx-exporter Debian packageoperations/puppetproduction+2 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Nov 30 2017, 3:30 PM
elukey moved this task from Backlog to Keep an eye on it on the User-Elukey board.Dec 18 2017, 9:20 AM
gerritbot subscribed.Jan 4 2018, 3:31 PM

Change 402069 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: use prometheus-jmx-exporter Debian package

https://gerrit.wikimedia.org/r/402069

gerritbot added a project: Patch-For-Review.Jan 4 2018, 3:31 PM

Change 402070 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: switch to using jmx-exporter jar from Debian package

https://gerrit.wikimedia.org/r/402070

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 5 2018, 9:45 AM
fgiunchedi moved this task from Doing to Backlog on the User-fgiunchedi board.Jan 16 2018, 11:17 AM
gerritbot added a comment.Feb 27 2018, 2:12 PM

Change 402069 merged by Elukey:
[operations/puppet@production] cassandra: use prometheus-jmx-exporter Debian package

https://gerrit.wikimedia.org/r/402069

gerritbot added a comment.Feb 27 2018, 2:27 PM

Change 415013 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cassandra: flip jmx exporter's jar reference

https://gerrit.wikimedia.org/r/415013

gerritbot added a comment.Feb 27 2018, 4:00 PM

Change 415013 merged by Elukey:
[operations/puppet@production] cassandra: flip jmx exporter's jar reference

https://gerrit.wikimedia.org/r/415013

elukey closed this task as Resolved.Feb 27 2018, 4:35 PM
elukey claimed this task.

Restbase and AQS' instances will pick up the new jar as part of the next round of reboots. I'd close this task and re-open if necessary.

elukey reopened this task as Open.Feb 27 2018, 4:36 PM

ETOOSOON, we'll need to cleanup the scap dirs on the hosts probably?

elukey moved this task from Keep an eye on it to In Progress on the User-Elukey board.Feb 28 2018, 3:22 PM
fgiunchedi added a comment.Feb 28 2018, 3:25 PM
In T181728#4006555, @elukey wrote:

ETOOSOON, we'll need to cleanup the scap dirs on the hosts probably?

Indeed!

elukey added a comment.Mar 27 2018, 1:02 PM

So lsof -X / | grep jmx_prometheus on restbase* shows only

java      17424        cassandra  mem    REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      17424        cassandra   65r   REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      19509        cassandra  mem    REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      19509        cassandra   65r   REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar

So I think that it should be safe to clear /srv/deployment/prometheus and close the task..

Stashbot subscribed.Apr 4 2018, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-04T12:02:37Z] <elukey> removing /srv/deployment/prometheus from restbase2001/1007 - T181728

elukey added a comment.Apr 4 2018, 12:04 PM

After removing /srv/deployment/prometheus I don't see any trace of the jmx exporter jar contained in the dir in lsof -Xd DEL on rdb2001/1007.

Stashbot added a comment.Apr 4 2018, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-04T15:06:22Z] <elukey> delete /srv/deployment/prometheus from restbase* as clean up step for T181728

elukey closed this task as Resolved.Apr 4 2018, 3:23 PM
gerritbot added a comment.Apr 27 2018, 2:27 PM

Change 402070 abandoned by Filippo Giunchedi:
cassandra: switch to using jmx-exporter jar from Debian package

Reason:
Not needed anymore, happened already

https://gerrit.wikimedia.org/r/402070

gerritbot mentioned this in T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)).Aug 11 2020, 8:05 AM
Stashbot added a comment.Aug 11 2020, 8:09 AM

Mentioned in SAL (#wikimedia-releng) [2020-08-11T08:09:40Z] <hashar> Gerrit: marked operations/software/prometheus_jmx_exporter read-only/archive since we now use a Debian package. T181728

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2020, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL