Deprecate cassandra-metrics-collector?
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Eevans
	Feb 5 2018, 10:12 PM

Description

The RESTBase cluster has been converted to Prometheus metrics, and there seems to be some interest in doing the same with AQS. The collector is currently setup as part of the profile since it has been the assumption that every Cassandra node should have it configured. We need to either a) convert every cluster to Prometheus and then remove cassandra-metrics-collector entirely, or b) refactor Puppet to make the choice of one or the other configurable.

In addition to AQS, there is a maps cluster that would need to be taken into account as well.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
cassandra: remove cassandra-metrics-collector	operations/puppet	production	+0 -203
maps: disable cassandra metrics collector	operations/puppet	production	+4 -0
cassandra::metrics: extend ensure to scap target	operations/puppet	production	+1 -0
RESTBase: Disable cassandra-metrics-collector	operations/puppet	production	+2 -66
cassandra::metrics: propagate ensure parameter to wmf-auto-restart	operations/puppet	production	+3 -1
role::aqs: deprecate cassandra-metrics-collector	operations/puppet	production	+27 -11

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T175344 Move away from jmxtrans in favor of prometheus jmx_exporter
Resolved	None	T186567 Deprecate cassandra-metrics-collector?
Resolved	elukey	T184795 Add the prometheus jmx agent to AQS Cassandra
Resolved	elukey	T189529 Test/upload new cassandra 2.2.6 package (wmf3)

Event Timeline

Eevans created this task.Feb 5 2018, 10:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 5 2018, 10:12 PM

I'm favor of deprecating cmc in favor of Prometheus, and going with a) as that seems simpler and the end state we want to be in anyway.

Eevans triaged this task as Medium priority.Feb 22 2018, 4:44 PM

To summarize a meeting w/ @Gehel :

Maps cluster is running Cassandra v2.1 (AQS runs v2.2 and RESTBase 3.11)
Maps cluster hasn't received much attention, but hasn't needed much either; Has been trouble-free
Currently no development resources allocated to making changes
Some metric names changed between v2.1 and v2.2, and again between v2.2 and v3.11
- Making dashboards work interchangeably for v2.2 and v3.11 clusters will require some work (may not be entirely practical)
- Making dashboards work interchangeably for v2.1, 2.2, and 3.11 clusters will require even more work (could be even less practical)

I propose that we first start with the AQS cluster by installing the Prometheus exporter and cassandra-metrics-collector side-by-side, and see where we stand with the dashboards. If it turns out to be straightforward to make the dashboards work interchangeably, then we can consider either a) doing the same with Maps, or b) upgrading Maps to Cassandra 2.2.

We should probably also reconsider a Puppet refactoring to make the choice of metrics collection configurable (since a quick transition is seeming less and less tractable).

Eevans added a subtask: T184795: Add the prometheus jmx agent to AQS Cassandra.Feb 22 2018, 9:13 PM

elukey closed subtask T189529: Test/upload new cassandra 2.2.6 package (wmf3) as Resolved.Mar 27 2018, 9:50 AM

In T186567#3993452, @Eevans wrote:

I propose that we first start with the AQS cluster by installing the Prometheus exporter and cassandra-metrics-collector side-by-side, and see where we stand with the dashboards. If it turns out to be straightforward to make the dashboards work interchangeably, then we can consider either a) doing the same with Maps, or b) upgrading Maps to Cassandra 2.2.

AQS is now running the jmx agent and its metrics are available in the prometheus analytics instance (not the services one). @Eevans I'd love to keep one set of dashboard shared between restbase and aqs/maps to avoid duplication of efforts, whenever you have time let's chat about the differences between 2.2 and 3.x.

In T186567#4083805, @elukey wrote:

In T186567#3993452, @Eevans wrote:

I propose that we first start with the AQS cluster by installing the Prometheus exporter and cassandra-metrics-collector side-by-side, and see where we stand with the dashboards. If it turns out to be straightforward to make the dashboards work interchangeably, then we can consider either a) doing the same with Maps, or b) upgrading Maps to Cassandra 2.2.

AQS is now running the jmx agent and its metrics are available in the prometheus analytics instance (not the services one). @Eevans I'd love to keep one set of dashboard shared between restbase and aqs/maps to avoid duplication of efforts, whenever you have time let's chat about the differences between 2.2 and 3.x.

I haven't looked too deeply, but if we could s/columnfamily/table/ the metric names, I think we'd be 99% of the way there, and without any calisthenics on the Grafana-side, (either via prometheus_jmx_exporter.yaml, or on the Prometheus server side of things).

• Nuria closed subtask T184795: Add the prometheus jmx agent to AQS Cassandra as Resolved.Mar 29 2018, 11:00 PM

fgiunchedi mentioned this in T191315: Cassandra Graphite metrics space usage audit and cleanup.Apr 3 2018, 4:15 PM

After T193017, the AQS cluster shares the same cassandra dashboards with Restbase. I tried this morning to disable the cassandra-metrics-collector on AQS (touching /etc/cassandra-metrics-collector/disable) but it doesn't seem to work due to:

the puppet class cassandra::metrics that hardcodes versions of the collectors to use

if $target_cassandra_version == '2.1' {
    $collector_version = '2.1.1-20160520.211019-1'
} elsif $target_cassandra_version == '2.2' {
    $collector_version = '3.1.4-20170427.001104-1'
} else {
    $collector_version = '4.1.0'
}

the jar related to cassandra-metrics-collector version 4.10 not present on AQS nodes (they are not in the scap deploy list of cassandra-metrics-collector).

While we wait for maps I think that a profile option to disable the metrics collector would be good for Restbase/AQS.

Change 431546 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::aqs: deprecate cassandra-metrics-collector

https://gerrit.wikimedia.org/r/431546

gerritbot added a project: Patch-For-Review.May 7 2018, 10:34 AM

Change 431546 merged by Elukey:
[operations/puppet@production] role::aqs: deprecate cassandra-metrics-collector

https://gerrit.wikimedia.org/r/431546

Change 431710 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cassandra::metrics: propagate ensure parameter to wmf-auto-restart

https://gerrit.wikimedia.org/r/431710

Change 431710 merged by Elukey:
[operations/puppet@production] cassandra::metrics: propagate ensure parameter to wmf-auto-restart

https://gerrit.wikimedia.org/r/431710

Cassandra metrics collector removed from AQS!

Change 444247 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] RESTBase: Disable cassandra-metrics-collector

https://gerrit.wikimedia.org/r/444247

Change 444247 merged by Filippo Giunchedi:
[operations/puppet@production] RESTBase: Disable cassandra-metrics-collector

https://gerrit.wikimedia.org/r/444247

This service is now disabledd/removed on the RESTBase cluster; Thanks a bunch for the assist!

However, the original objective (the deprecation of the collector) remains unresolved until (if?) the maps cluster is upgraded to Cassandra 2.2.x. :(

• mobrovac added a project: Platform Team Legacy (Later).Dec 20 2018, 12:17 PM

elukey@maps1001:~$ dpkg -l | grep cassandra
ii  cassandra                            2.2.6-wmf5                        all          distributed storage system for structured data
ii  cassandra-tools                      2.2.6-wmf5                        all          distributed storage system for structured data
ii  cassandra-tools-wmf                  1.0.2-1                           all          add-ons to make Wikimedia Cassandra operations easier
ii  python-cassandra                     3.7.1-2.1                         amd64        Python driver for Apache Cassandra

@Eevans I re-discovered this task while trying to deploy AQS in Cloud without a deployment server, since it failed due cassandra::metrics (even if absented it tries to ensure some scap config). Maps seems to be on 2.2 now, maybe we could think about moving the cluster as we did for AQS?

Change 662634 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cassandra::metrics: extend ensure to scap target

https://gerrit.wikimedia.org/r/662634

Change 662634 abandoned by Elukey:
[operations/puppet@production] cassandra::metrics: extend ensure to scap target

Reason:

https://gerrit.wikimedia.org/r/662634

Eevans removed a project: User-Eevans.Jun 9 2021, 4:44 PM

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2021, 5:11 PM

hnowlan added a project: Platform Team Workboards (Platform Engineering Reliability).Aug 4 2021, 4:05 PM

Change 710984 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: disable cassandra metrics collector

https://gerrit.wikimedia.org/r/710984

gerritbot added a project: Patch-For-Review.Aug 9 2021, 3:47 PM

Change 710985 had a related patch set uploaded (by Hnowlan; author: Hnowlan):