Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Oct 7 2020, 6:11 PM

Description

The dashboard at https://grafana-labs.wikimedia.org/d/000000059/cloud-vps-project-board is currently the most comprehensive way to look at the system health across a Cloud VPS project. This dashboard is using the legacy Graphite data collected by Diamond rather than the newer Prometheus data collection system.

T210993: Deprecate Diamond collectors in Cloud VPS has been working towards the removal of Diamond collectors for quite some time and https://gerrit.wikimedia.org/r/c/operations/puppet/+/632471 is currently proposed to complete the removal. When this happens the related Graphite data will stop being collected and the dashboard will become stale.

Related Objects
Search...

Status	Assigned	Task
Resolved	fgiunchedi	T205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal)
Resolved	colewhite	T183454 Deprovision Diamond collectors no longer in use
Resolved	MoritzMuehlenhoff	T210993 Deprecate Diamond collectors in Cloud VPS
Resolved	taavi	T336774 Current status of cloudmetrics and its components
Resolved	taavi	T326266 Remove the WMCS statsd/Graphite service
Resolved	dcaro	T313444 Streamline WMCS Alerting and Paging
Resolved	taavi	T317032 Remove Diamond?
Resolved	taavi	T264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
Resolved	taavi	T266050 Build Prometheus service for use by all Cloud VPS projects and their instances
Resolved	• JHedden	T250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
Resolved	• JHedden	T250210 Request creation of metricsinfra VPS project
Resolved	• Bstorm	T252260 Try out removing the nfs client probes from the node exporter on VMs
Resolved	bd808	T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
Resolved	Andrew	T128615 Get rid of Toolforge home page check from shinken
Declined	None	T128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
Resolved	• nskaggs	T284973 Request increased quota for metricsinfra Cloud VPS project
Resolved	taavi	T285055 Allow project members/admins to ack/silence alerts of that project
Resolved	jbond	T286716 Create solution for developer account authentication for services hosted in Cloud VPS
Resolved	taavi	T286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
Resolved	taavi	T286335 Split metricsinfra alertmanager to separate hosts from prometheus
Resolved	taavi	T287148 Set up IRC relay for metricsinfra alerts
Resolved	taavi	T310799 Upgrade metricsinfra prometheus to bullseye
Resolved	dcaro	T310802 Request increased quota for metricsinfra Cloud VPS project
Resolved	Andrew	T288108 Figure out how to deal with security groups when rolling out metricsinfra scraping

Event Timeline

bd808 created this task.Oct 7 2020, 6:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2020, 6:11 PM

bd808 added a subtask: T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.Oct 7 2020, 6:22 PM

• nskaggs moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.Oct 20 2020, 4:29 PM

bd808 mentioned this in T266050: Build Prometheus service for use by all Cloud VPS projects and their instances.Oct 20 2020, 5:02 PM

bd808 edited subtasks, added: T266050: Build Prometheus service for use by all Cloud VPS projects and their instances; removed: T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.

bd808 added a parent task: T210993: Deprecate Diamond collectors in Cloud VPS.

Dzahn mentioned this in T210993: Deprecate Diamond collectors in Cloud VPS.Nov 17 2020, 1:10 AM

The code at https://gerrit.wikimedia.org/r/c/operations/puppet/+/632570/3/modules/profile/manifests/wmcs/instance.pp talks about checking "puppet freshness".

The dashboard linked here https://grafana-labs.wikimedia.org/d/000000059/cloud-vps-project-board?orgId=1 has some "Puppet agent" but only 2 and they have no data?

Is that the same thing or unrelated?

Because if those puppet freshness checks don't exist or are not used we could maybe still remove that part from instances.

In T264920#6665436, @Dzahn wrote:

Is that the same thing or unrelated?

See T266050: Build Prometheus service for use by all Cloud VPS projects and their instances. TL;DR we need to build a Prometheus cluster that can scrape all ~700 Cloud VPS instances to move the puppet freshness dashboard to Prometheus backed data.

Ok, thanks. Gotcha.

Andrew triaged this task as Medium priority.Jan 12 2021, 5:08 PM

• nnikkhoui subscribed.Sep 14 2021, 9:23 PM

Dzahn mentioned this in T272559: Unused puppet resources audit, 2021.Dec 3 2021, 6:56 PM