Page MenuHomePhabricator
Create Task
Maniphest T192456

Prometheus metrics missing for some hosts
Closed, ResolvedPublic
Actions

Description

The RESTBase Cassandra cluster has some nodes where Prometheus metrics (those that come from the JMX exporter), are missing.

Related Objects

Event Timeline

Eevans created this task.Apr 18 2018, 2:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 18 2018, 2:36 PM
Eevans triaged this task as High priority.Apr 18 2018, 2:36 PM
Stashbot subscribed.Apr 18 2018, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-18T14:37:00Z] <urandom> restarting Cassandra, restbase1011-a -- T192456

Stashbot added a comment.Apr 18 2018, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-18T14:55:19Z] <urandom> restarting Cassandra, restbase1011-a to test v 0.8 of Prometheus JMX exporter -- T192456

Eevans added a subscriber: elukey.Apr 18 2018, 3:08 PM

For the machines affected, executing curl against the exporter URL just hangs indefinitely. I attempted to restart 1011-a to no avail. I then live-hacked cassandra-env.sh to roll back the exporter jar to the 0.8 version we used before, and it is now working. More investigation is needed.

Stashbot mentioned this in T189822: Replace 5 Samsung SSD 850 devices w/ 4 1.6T Intel or HP SSDs.Apr 19 2018, 8:48 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-19T20:48:12Z] <urandom> restarting cassandra to (temporarily) rollback prometheus jmx exporter -- T189822, T192456

Stashbot added a comment.Apr 19 2018, 8:48 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-19T20:48:24Z] <urandom> restarting cassandra to (temporarily) rollback prometheus jmx exporter, restbase1010-a -- T189822, T192456

Stashbot added a comment.Apr 19 2018, 9:12 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-19T21:11:56Z] <urandom> restarting cassandra to (temporarily) rollback prometheus jmx exporter, restbase1010-c -- T189822, T192456

fgiunchedi mentioned this in T192768: wdqs-updater crashing not cleanly.Apr 24 2018, 8:44 AM
Eevans closed this task as Resolved.Jul 3 2018, 10:09 PM

This was resolved by the upgrade to 1:0.3.0

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL