JMX exporter for cassandra is enabled in the dev cluster now, we should provision a new prometheus instance in codfw and eqiad (named services) to collect such metrics
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Eevans | T160570 Cassandra 3.x Tracking | |||
| Resolved | Eevans | T169936 Services 2017/18 Q1 goal: Start gradual roll-out of Cassandra 3 & new schema to resolve storage scaling issues and OOM errors. | |||
| Resolved | Eevans | T171772 Prometheus metrics storage for RESTBase dev environment | |||
| Resolved | fgiunchedi | T173490 Provision prometheus instance for cassandra/services metrics collection |
Event Timeline
Comment Actions
Change 372357 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP: new prometheus instance 'services'
Comment Actions
Change 372357 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: new instance 'services'
Comment Actions
Prometheus instance is up and running, still missing the "targets" generation, i.e. the cassandra instances that are currently running jmx_exporter.
Comment Actions
I can't seem to get the following to work to extract all hosts that have prometheus::jmx_exporter_instance defined:
root@puppetmaster1001:~# puppet apply -e 'notice(query_resources(false, "Define[\"prometheus::jmx_exporter_instance\"]", false))' Notice: Scope(Class[main]):
Comment Actions
Change 372845 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] role: collect from restbase test_cluster
Comment Actions
Change 372845 merged by Filippo Giunchedi:
[operations/puppet@production] role: collect jmx_exporter metrics from restbase test_cluster
Comment Actions
This is done, the services Prometheus instance is up and running and collecting cassandra metrics