Page MenuHomePhabricator
Create Task
Maniphest T336774

Current status of cloudmetrics and its components
Closed, ResolvedPublic
Actions

Description

This task captures the current (May 2023) status of cloudmetrics hosts and their components, and opens the discussion on what their future might be. Please feel free to edit the description if I (@fgiunchedi) missed something.

There are two cloudmetrics hosts, each running the following:

  1. graphite/statsd
  2. grafana
  3. prometheus

Given the big progress that's happened lately (on a multiple-months scale that is, also thanks to the excellent work by @taavi) with metricsinfra project, I'd like to pose the following questions:

  1. Is graphite/statsd on cloudmetrics still actively consumed/useful? (i.e. does anyone look at the metrics?)
  2. Ditto for grafana on cloudmetrics? (i.e. https://grafana-cloud.wikimedia.org). I believe the answer is likely "on its way out" given T333568: Move WMCS dashboards to grafana.wmcloud.org

Ideally both answers are 'no', but regardless I'd like to move the labs instance of Prometheus off cloudmetrics and onto the Prometheus production hardware. My understanding is that this move will help both WMCS (one less component to think about) and Observability (less variance/snowflakes).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove cloudmetrics Cumin aliasoperations/puppetproduction+0 -2
O:wmcs::monitoring: drop roleoperations/puppetproduction+0 -88
site: Re-image cloudmetrics hosts as insetupoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedtaaviT336774 Current status of cloudmetrics and its components
 ResolvedtaaviT326266 Remove the WMCS statsd/Graphite service
 ResolvedtaaviT317032 Remove Diamond?
 ResolvedtaaviT264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
 ResolvedtaaviT266050 Build Prometheus service for use by all Cloud VPS projects and their instances
 Resolved• JHeddenT250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
 Resolved• JHeddenT250210 Request creation of metricsinfra VPS project
 Resolved• BstormT252260 Try out removing the nfs client probes from the node exporter on VMs
 Resolvedbd808T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
 ResolvedAndrewT128615 Get rid of Toolforge home page check from shinken
 DeclinedNoneT128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
 Resolved• nskaggsT284973 Request increased quota for metricsinfra Cloud VPS project
 ResolvedtaaviT285055 Allow project members/admins to ack/silence alerts of that project
 ResolvedjbondT286716 Create solution for developer account authentication for services hosted in Cloud VPS
 ResolvedtaaviT286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
 ResolvedtaaviT286335 Split metricsinfra alertmanager to separate hosts from prometheus
 ResolvedtaaviT287148 Set up IRC relay for metricsinfra alerts
 ResolvedtaaviT310799 Upgrade metricsinfra prometheus to bullseye
 ResolveddcaroT310802 Request increased quota for metricsinfra Cloud VPS project
 ResolvedAndrewT288108 Figure out how to deal with security groups when rolling out metricsinfra scraping
 ResolvedhasharT348830 Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud
ResolvedfgiunchediT336854 Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts
 ResolvedtaaviT349801 Linting problems found for OpenstackAPIResponse
 ResolvedfgiunchediT350010 Deploy 'cloud' Prometheus instance to codfw
 OpenNoneT350013 Rename prometheus/labs datasource in Grafana to prometheus/cloud

Event Timeline

fgiunchedi created this task.May 16 2023, 3:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2023, 3:13 PM
fgiunchedi added subscribers: • aborrero, dcaro.May 16 2023, 3:14 PM
fgiunchedi updated the task description. (Show Details)May 16 2023, 3:17 PM
fgiunchedi added a subscriber: fnegri.
taavi edited projects, added Cloud-VPS, cloud-services-team; removed Cloud-Services.May 16 2023, 3:27 PM
taavi added subscribers: Andrew, bd808.

Hi!

Is graphite/statsd on cloudmetrics still actively consumed/useful? (i.e. does anyone look at the metrics?)

Not really[1]. There are some grafana-cloud dashboards (and a majority of those are probably unused these days) using it but the equivalent metrics are present on the metricsinfra Prometheus instance.

[1]: Excluding deployment-prep, which is a whole different beast. It's been broken for months already due to hostname changes so I don't think it's a blocker. T241285.

Ditto for grafana on cloudmetrics? (i.e. https://grafana-cloud.wikimedia.org). I believe the answer is likely "on its way out" given T333568: Move WMCS dashboards to grafana.wmcloud.org

Indeed the plan is to replace it with https://grafana.wmcloud.org and https://grafana.wikimedia.org as discussed in T307465: grafana-cloud: Browser access to Prometheus is deprecated.

For both of these, I think a reasonable plan of action would be the following:

  1. Finish migrating WMCS-managed dashboards off grafana-cloud: T333568
  2. Pick some dates when this will happen, and send a cloud-announce@ notification about these changes.
  3. Disable Diamond on the remaining Cloud VPS nodes. This is a one-line hiera change.
  4. Shut down the graphite and statsd services on cloudmetrics nodes.
  5. Export a static dump of the Grafana dashboard configurations, and put it up somewhere (download.wmcloud.org?)
  6. Shut down the Grafana server on cloudmetrics nodes. Redirect it to a Wikitech page explaining where to find the dashboards now.

I've been meaning to do it on my own but haven't yet gotten to it. Wdyt?

See also:

Ideally both answers are 'no', but regardless I'd like to move the labs instance of Prometheus off cloudmetrics and onto the Prometheus production hardware. My understanding is that this move will help both WMCS (one less component to think about) and Observability (less variance/snowflakes).

This sounds reasonable to me. I'd like to use this opportunity to get an equivalent Prometheus instance for the codfw1dev testing deployment which does not have cloudmetrics hardware of its own.

/cc @Andrew @bd808 in case you have comments about the plan above.

fgiunchedi added a comment.May 17 2023, 7:51 AM
In T336774#8855896, @taavi wrote:

Hi!

Is graphite/statsd on cloudmetrics still actively consumed/useful? (i.e. does anyone look at the metrics?)

Not really[1]. There are some grafana-cloud dashboards (and a majority of those are probably unused these days) using it but the equivalent metrics are present on the metricsinfra Prometheus instance.

[1]: Excluding deployment-prep, which is a whole different beast. It's been broken for months already due to hostname changes so I don't think it's a blocker. T241285.

Thank you for the context, I'm happy to know metricsinfra has equivalent metrics! And agreed re: deployment-prep, better to followup on T241285 in case.

Ditto for grafana on cloudmetrics? (i.e. https://grafana-cloud.wikimedia.org). I believe the answer is likely "on its way out" given T333568: Move WMCS dashboards to grafana.wmcloud.org

Indeed the plan is to replace it with https://grafana.wmcloud.org and https://grafana.wikimedia.org as discussed in T307465: grafana-cloud: Browser access to Prometheus is deprecated.

For both of these, I think a reasonable plan of action would be the following:

  1. Finish migrating WMCS-managed dashboards off grafana-cloud: T333568
  2. Pick some dates when this will happen, and send a cloud-announce@ notification about these changes.
  3. Disable Diamond on the remaining Cloud VPS nodes. This is a one-line hiera change.
  4. Shut down the graphite and statsd services on cloudmetrics nodes.
  5. Export a static dump of the Grafana dashboard configurations, and put it up somewhere (download.wmcloud.org?)
  6. Shut down the Grafana server on cloudmetrics nodes. Redirect it to a Wikitech page explaining where to find the dashboards now.

I've been meaning to do it on my own but haven't yet gotten to it. Wdyt?

The plan looks good to me! I'm happy to assist with reviews, merges, etc

See also:

Ideally both answers are 'no', but regardless I'd like to move the labs instance of Prometheus off cloudmetrics and onto the Prometheus production hardware. My understanding is that this move will help both WMCS (one less component to think about) and Observability (less variance/snowflakes).

This sounds reasonable to me. I'd like to use this opportunity to get an equivalent Prometheus instance for the codfw1dev testing deployment which does not have cloudmetrics hardware of its own.

Yes that'd be doable, essentially we would deploy a wmcs Prometheus instance (might as well rename while we're at it) in both eqiad and codfw, each pulling from their respective openstack deployment.

• aborrero added a comment.May 17 2023, 10:11 AM
In T336774#8858089, @fgiunchedi wrote:

Yes that'd be doable, essentially we would deploy a wmcs Prometheus instance (might as well rename while we're at it) in both eqiad and codfw, each pulling from their respective openstack deployment.

I agree with this plan and with everything commented above.

thanks you both!

taavi added a subtask: T326266: Remove the WMCS statsd/Graphite service.May 17 2023, 12:20 PM
taavi mentioned this in T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts.May 17 2023, 12:25 PM
dcaro added a comment.May 17 2023, 12:34 PM

+1 for the plan too

MoritzMuehlenhoff subscribed.May 17 2023, 2:48 PM
• lmata subscribed.May 17 2023, 5:33 PM
fgiunchedi added a project: User-fgiunchedi.Jun 1 2023, 1:26 PM
taavi added a comment.Jun 29 2023, 3:54 PM
In T336774#8855896, @taavi wrote:
  1. Pick some dates when this will happen, and send a cloud-announce@ notification about these changes.

In the interest of moving this forward I've come up with the following dates rather arbitrarily:

  • Disable Diamond everywhere on (or slightly after) Monday, July 17th (in about two weeks' time)
  • Disable access to Graphite and the legacy Grafana instance on Tuesday, August 1st (in about a month)

The logic here is that as we have data equivalent to what Diamond produces so migrating those should be fairly simple, while there might be some more complicated Graphite use cases that we want to give time to migrate (especially as people might be on vacations etc during these months)

I will send announcements on Monday (July 3rd) unless there are major objections.

fgiunchedi added a comment.Jun 30 2023, 8:00 AM

FWIW the timeline looks good to me, thank you @taavi

gerritbot added a comment.Oct 24 2023, 2:31 PM

Change 968277 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] site: Re-image cloudmetrics hosts as insetup

https://gerrit.wikimedia.org/r/968277

gerritbot added a project: Patch-For-Review.Oct 24 2023, 2:31 PM

Change 968282 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] O:wmcs::monitoring: drop role

https://gerrit.wikimedia.org/r/968282

fgiunchedi removed a project: User-fgiunchedi.Oct 27 2023, 6:56 AM
gerritbot added a comment.Oct 27 2023, 7:22 AM

Change 968277 merged by Majavah:

[operations/puppet@production] site: Re-image cloudmetrics hosts as insetup

https://gerrit.wikimedia.org/r/968277

ops-monitoring-bot added a comment.Oct 27 2023, 7:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by taavi@cumin1001 for host cloudmetrics1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 27 2023, 7:54 AM

Cookbook cookbooks.sre.hosts.reimage started by taavi@cumin1001 for host cloudmetrics1003.eqiad.wmnet with OS bookworm completed:

  • cloudmetrics1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310270739_taavi_1264680_cloudmetrics1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Oct 27 2023, 7:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by taavi@cumin1001 for host cloudmetrics1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 27 2023, 8:25 AM

Cookbook cookbooks.sre.hosts.reimage started by taavi@cumin1001 for host cloudmetrics1004.eqiad.wmnet with OS bookworm completed:

  • cloudmetrics1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310270810_taavi_1282198_cloudmetrics1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 27 2023, 9:15 AM

Change 968282 merged by Majavah:

[operations/puppet@production] O:wmcs::monitoring: drop role

https://gerrit.wikimedia.org/r/968282

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2023, 9:30 AM
taavi closed subtask T326266: Remove the WMCS statsd/Graphite service as Resolved.Oct 27 2023, 9:32 AM
gerritbot added a comment.Oct 27 2023, 10:07 AM

Change 969309 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Remove cloudmetrics Cumin alias

https://gerrit.wikimedia.org/r/969309

gerritbot added a project: Patch-For-Review.Oct 27 2023, 10:07 AM
gerritbot added a comment.Oct 27 2023, 10:10 AM

Change 969309 merged by Majavah:

[operations/puppet@production] Remove cloudmetrics Cumin alias

https://gerrit.wikimedia.org/r/969309

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2023, 10:10 AM
fgiunchedi closed subtask T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts as Resolved.Oct 30 2023, 10:56 AM
ops-monitoring-bot added a comment.Nov 13 2023, 11:01 AM

cookbooks.sre.hosts.decommission executed by taavi@cumin1001 for hosts: cloudmetrics[1003-1004].eqiad.wmnet

  • cloudmetrics1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • cloudmetrics1004.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
dcaro mentioned this in T362452: Taavi knowledge transfer: cloud-vps monitoring.Apr 24 2024, 9:26 AM
taavi closed this task as Resolved.Jun 4 2024, 4:01 PM
taavi claimed this task.
fgiunchedi closed subtask T350010: Deploy 'cloud' Prometheus instance to codfw as Resolved.Aug 21 2024, 10:26 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits