Page MenuHomePhabricator
Create Task
Maniphest T240715

Configure prometheus monitoring for Ceph
Closed, ResolvedPublic
Actions

Description

The Ceph manager has a plugin that exposes metrics to prometheus. https://docs.ceph.com/docs/master/mgr/prometheus/

Ceph manager is configured to run on all 3 monitor hosts (cloudcephmon100[1-3]). However, it's only active on one host at any given time. We'll need to configure a load balancer with all 3 backend cloudcephmon100[1-3] hosts on TCP port 9283. The current version of Ceph allows you to add all manager hosts to the prometheus scrape target.

  • Enable the prometheus plugin
  • Build / Find grafana dashboard
  • Identify metrics for alerting

Details

SubjectRepoBranchLines +/-
icinga: add cloudvps ceph alertsoperations/puppetproduction+44 -0
ceph: add wmcs prometheus scrape configoperations/puppetproduction+39 -0
ceph: add prometheus scrape configoperations/puppetproduction+18 -1
lvs: update cloudceph proxy check urloperations/puppetproduction+1 -1
ceph: add prometheus servers to ferm rulesoperations/puppetproduction+12 -10
ceph: update ferm for prometheus exporteroperations/puppetproduction+12 -10
ceph: Update ceph role desc and lvs pool mapoperations/puppetproduction+6 -4
lvs ceph: add cloudceph service and clusteroperations/puppetproduction+33 -0
ceph: allow lvs traffic to manager exporteroperations/puppetproduction+9 -0
add forward and reverse for cloudceph.svc.eqiad.wmnetoperations/dnsmaster+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• JHedden created this task.Dec 13 2019, 8:11 PM
• JHedden added a comment.Dec 13 2019, 10:24 PM

grafana dashboards that work with the ceph prometheus plugin can be found at https://github.com/ceph/ceph/tree/master/monitoring/grafana

• JHedden updated the task description. (Show Details)Dec 17 2019, 10:15 PM
• JHedden updated the task description. (Show Details)Dec 17 2019, 10:18 PM
gerritbot added a comment.Dec 17 2019, 10:31 PM

Change 558707 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/dns@master] add forward and reverse for cloudcephmgr.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/558707

gerritbot added a project: Patch-For-Review.Dec 17 2019, 10:31 PM
gerritbot added a comment.Dec 18 2019, 3:54 PM

Change 559110 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] lvs ceph: add cloudceph service and cluster

https://gerrit.wikimedia.org/r/559110

gerritbot added a comment.Dec 20 2019, 10:02 PM

Change 558707 merged by Jhedden:
[operations/dns@master] add forward and reverse for cloudceph.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/558707

gerritbot added a comment.Dec 23 2019, 4:17 PM

Change 560410 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: allow lvs traffic to manager exporter

https://gerrit.wikimedia.org/r/560410

gerritbot added a comment.Dec 23 2019, 4:24 PM

Change 560410 merged by Jhedden:
[operations/puppet@production] ceph: allow lvs traffic to manager exporter

https://gerrit.wikimedia.org/r/560410

gerritbot added a comment.Jan 7 2020, 5:05 PM

Change 559110 merged by Jhedden:
[operations/puppet@production] lvs ceph: add cloudceph service and cluster

https://gerrit.wikimedia.org/r/559110

Stashbot added a comment.Jan 7 2020, 5:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-07T17:13:22Z] <vgutierrez> restarting pybal on lvs1016 - T240715

Stashbot added a comment.Jan 7 2020, 5:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-07T17:18:04Z] <vgutierrez> restarting pybal on lvs1015 - T240715

gerritbot added a comment.Jan 7 2020, 6:26 PM

Change 562574 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: Update ceph role system::role name

https://gerrit.wikimedia.org/r/562574

gerritbot added a comment.Jan 7 2020, 7:26 PM

Change 562574 merged by Jhedden:
[operations/puppet@production] ceph: Update ceph role desc and lvs pool map

https://gerrit.wikimedia.org/r/562574

gerritbot added a comment.Jan 7 2020, 8:39 PM

Change 562610 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: update ferm for prometheus exporter

https://gerrit.wikimedia.org/r/562610

gerritbot added a comment.Jan 7 2020, 8:41 PM

Change 562610 merged by Jhedden:
[operations/puppet@production] ceph: update ferm for prometheus exporter

https://gerrit.wikimedia.org/r/562610

gerritbot added a comment.Jan 7 2020, 9:00 PM

Change 562616 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: add prometheus servers to ferm rules

https://gerrit.wikimedia.org/r/562616

gerritbot added a comment.Jan 7 2020, 9:02 PM

Change 562616 merged by Jhedden:
[operations/puppet@production] ceph: add prometheus servers to ferm rules

https://gerrit.wikimedia.org/r/562616

• JHedden moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jan 7 2020, 10:24 PM
gerritbot added a comment.Jan 7 2020, 10:52 PM

Change 562637 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] lvs: update cloudceph proxy check url

https://gerrit.wikimedia.org/r/562637

gerritbot added a comment.Jan 8 2020, 10:35 PM

Change 562979 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: add prometheus scrape config

https://gerrit.wikimedia.org/r/562979

gerritbot added a comment.Jan 8 2020, 10:36 PM

Change 562637 abandoned by Jhedden:
lvs: update cloudceph proxy check url

https://gerrit.wikimedia.org/r/562637

• JHedden updated the task description. (Show Details)Jan 8 2020, 10:36 PM
• JHedden updated the task description. (Show Details)Jan 8 2020, 10:39 PM
gerritbot added a comment.Jan 9 2020, 2:39 PM

Change 562979 abandoned by Jhedden:
ceph: add prometheus scrape config

https://gerrit.wikimedia.org/r/562979

gerritbot added a comment.Jan 9 2020, 3:09 PM

Change 563190 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: add wmcs prometheus scrape config

https://gerrit.wikimedia.org/r/563190

gerritbot added a comment.Jan 9 2020, 3:57 PM

Change 563190 merged by Jhedden:
[operations/puppet@production] ceph: add wmcs prometheus scrape config

https://gerrit.wikimedia.org/r/563190

• JHedden updated the task description. (Show Details)Jan 9 2020, 11:16 PM
• JHedden closed this task as Resolved.Jan 15 2020, 3:57 PM
• JHedden triaged this task as Medium priority.
• JHedden updated the task description. (Show Details)
gerritbot added a comment.Feb 14 2020, 9:25 PM

Change 572355 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] icinga: add cloudvps ceph alerts

https://gerrit.wikimedia.org/r/572355

gerritbot added a comment.Feb 14 2020, 9:46 PM

Change 572355 merged by Dzahn:
[operations/puppet@production] icinga: add cloudvps ceph alerts

https://gerrit.wikimedia.org/r/572355

Dzahn subscribed.Feb 14 2020, 9:51 PM

new checks have been added to Icinga:

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits