Page MenuHomePhabricator

Stop using jmx_exporter deployed via scap in favour of Debian package
Closed, ResolvedPublic

Description

jmx_exporter is packaged as Debian (by us) prometheus-jmx-exporter, we should switch current users of jmx_exporter deployed via scap (only cassandra-restbase afaik) to use the Debian package instead.

Details

Related Gerrit Patches:

Event Timeline

elukey moved this task from Backlog to Keep an eye on it on the User-Elukey board.Dec 18 2017, 9:20 AM

Change 402069 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: use prometheus-jmx-exporter Debian package

https://gerrit.wikimedia.org/r/402069

Change 402070 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: switch to using jmx-exporter jar from Debian package

https://gerrit.wikimedia.org/r/402070

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 5 2018, 9:45 AM
fgiunchedi moved this task from Doing to Backlog on the User-fgiunchedi board.Jan 16 2018, 11:17 AM

Change 402069 merged by Elukey:
[operations/puppet@production] cassandra: use prometheus-jmx-exporter Debian package

https://gerrit.wikimedia.org/r/402069

Change 415013 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cassandra: flip jmx exporter's jar reference

https://gerrit.wikimedia.org/r/415013

Change 415013 merged by Elukey:
[operations/puppet@production] cassandra: flip jmx exporter's jar reference

https://gerrit.wikimedia.org/r/415013

elukey closed this task as Resolved.Feb 27 2018, 4:35 PM
elukey claimed this task.

Restbase and AQS' instances will pick up the new jar as part of the next round of reboots. I'd close this task and re-open if necessary.

elukey reopened this task as Open.Feb 27 2018, 4:36 PM

ETOOSOON, we'll need to cleanup the scap dirs on the hosts probably?

ETOOSOON, we'll need to cleanup the scap dirs on the hosts probably?

Indeed!

So lsof -X / | grep jmx_prometheus on restbase* shows only

java      17424        cassandra  mem    REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      17424        cassandra   65r   REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      19509        cassandra  mem    REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar
java      19509        cassandra   65r   REG    9,0   356079    1015 /usr/share/java/prometheus/jmx_prometheus_javaagent-0.10.jar

So I think that it should be safe to clear /srv/deployment/prometheus and close the task..

Mentioned in SAL (#wikimedia-operations) [2018-04-04T12:02:37Z] <elukey> removing /srv/deployment/prometheus from restbase2001/1007 - T181728

After removing /srv/deployment/prometheus I don't see any trace of the jmx exporter jar contained in the dir in lsof -Xd DEL on rdb2001/1007.

Mentioned in SAL (#wikimedia-operations) [2018-04-04T15:06:22Z] <elukey> delete /srv/deployment/prometheus from restbase* as clean up step for T181728

elukey closed this task as Resolved.Apr 4 2018, 3:23 PM

Change 402070 abandoned by Filippo Giunchedi:
cassandra: switch to using jmx-exporter jar from Debian package

Reason:
Not needed anymore, happened already

https://gerrit.wikimedia.org/r/402070