Page MenuHomePhabricator
Create Task
Maniphest T193017

Unify, if possible, AQS and Restbase's cassandra dashboards
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

AQS now runs the prometheus jmx exporter for cassandra metrics, so it would be really good to be able to share the restbase's cassandra dashboard without re-inventing another one (same thing would be good for the maps cluster).

Currently there is a problem with metric names that changed from 2.x to 3.x, and Filippo suggested a rename via prometheus masters as workaround:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cmetric_relabel_configs%3E

Details

SubjectRepoBranchLines +/-
prometheus: enable metrics relabeloperations/puppetproduction+20 -0
role::prometheus::analytics: fix cassandra relabel configoperations/puppetproduction+1 -2
role::prometheus::analytics: rename cassandra metrics/labelsoperations/puppetproduction+18 -0
cassandra: add percentile metrics to 2.x's prometheus jmx configoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects

Event Timeline

elukey triaged this task as Medium priority.Apr 25 2018, 12:29 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 25 2018, 12:29 PM
elukey added a comment.Apr 25 2018, 2:10 PM

I created https://grafana-admin.wikimedia.org/dashboard/db/cassandra-aqs to port manually all the metrics names and see the discrepancies. This is what I found so far:

  • cassandra_columnfamily_* metrics have been renamed to cassandra_table_* in 3.x (same thing with labels named "columnfamily", that now are "table").
  • Percentile metrics like cassandra_table_readlatency_75p seems not present in 2.x
gerritbot subscribed.Apr 25 2018, 2:40 PM

Change 428931 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cassandra: add percentile metrics to 2.2's prometheus jmx config

https://gerrit.wikimedia.org/r/428931

gerritbot added a project: Patch-For-Review.Apr 25 2018, 2:40 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 25 2018, 2:43 PM
gerritbot added a comment.Apr 25 2018, 4:36 PM

Change 428931 merged by Elukey:
[operations/puppet@production] cassandra: add percentile metrics to 2.x's prometheus jmx config

https://gerrit.wikimedia.org/r/428931

elukey added a comment.Apr 25 2018, 5:11 PM

Cloned all the Cassandra dashboards created by the Services team, and adapted to AQS. The only change that I had to do was to replace 'table' with 'columnfamily'.

https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-tables -> Partition size seems still not working, those metrics are missing (only two, not a big deal).

• fdans lowered the priority of this task from Medium to Low.Apr 30 2018, 4:33 PM
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.
gerritbot added a comment.May 2 2018, 3:29 PM

Change 430399 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::analytics: rename cassandra metrics/labels

https://gerrit.wikimedia.org/r/430399

elukey moved this task from Backlog to In Progress on the User-Elukey board.May 3 2018, 5:52 AM
gerritbot added a comment.May 3 2018, 8:27 AM

Change 430399 merged by Elukey:
[operations/puppet@production] role::prometheus::analytics: rename cassandra metrics/labels

https://gerrit.wikimedia.org/r/430399

gerritbot added a comment.May 3 2018, 9:03 AM

Change 430563 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::analytics: fix cassandra relabel config

https://gerrit.wikimedia.org/r/430563

gerritbot added a comment.May 3 2018, 9:06 AM

Change 430563 merged by Elukey:
[operations/puppet@production] role::prometheus::analytics: fix cassandra relabel config

https://gerrit.wikimedia.org/r/430563

elukey added a comment.May 3 2018, 9:18 AM

Relabeling/naming done! These dashboards are now shared between Services and Analytics:

https://grafana.wikimedia.org/dashboard/db/cassandra
https://grafana.wikimedia.org/dashboard/db/cassandra-client-request?orgId=1
https://grafana.wikimedia.org/dashboard/db/cassandra-read-repair?orgId=1
https://grafana.wikimedia.org/dashboard/db/cassandra-system?orgId=1
https://grafana.wikimedia.org/dashboard/db/cassandra-tables?orgId=1
https://grafana.wikimedia.org/dashboard/db/cassandra-threadpools?orgId=1

elukey set the point value for this task to 5.May 3 2018, 9:18 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
elukey moved this task from In Progress to Done on the User-Elukey board.May 7 2018, 9:45 AM
elukey mentioned this in T186567: Deprecate cassandra-metrics-collector?.May 7 2018, 10:16 AM
Gehel mentioned this in T194997: Track more detailed disk usage on maps servers.May 22 2018, 4:00 PM
• Nuria closed this task as Resolved.Jun 11 2018, 10:38 PM
• Vvjjkkii renamed this task from Unify, if possible, AQS and Restbase's cassandra dashboards to q9daaaaaaa.Jul 1 2018, 1:13 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed elukey as the assignee of this task.
• Vvjjkkii raised the priority of this task from Low to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed the point value for this task.
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
Mainframe98 renamed this task from q9daaaaaaa to Unify, if possible, AQS and Restbase's cassandra dashboards.Jul 1 2018, 8:24 AM
Mainframe98 closed this task as Resolved.
Mainframe98 assigned this task to elukey.
Mainframe98 lowered the priority of this task from High to Low.
Mainframe98 removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Mainframe98 updated the task description. (Show Details)
Mainframe98 set the point value for this task to 5.
Mainframe98 added subscribers: gerritbot, Aklapper.
gerritbot added a comment.May 9 2019, 3:06 AM

Change 508809 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] prometheus: enable metrics relabel

https://gerrit.wikimedia.org/r/508809

gerritbot added a comment.May 14 2019, 7:59 AM

Change 508809 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: enable metrics relabel

https://gerrit.wikimedia.org/r/508809

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL