Page MenuHomePhabricator
Create Task
Maniphest T266050

Build Prometheus service for use by all Cloud VPS projects and their instances
Closed, ResolvedPublic
Actions

Description

In T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken we worked on a proof of concept project to design a replacement for monitoring a subset of projects using Shinken with Prometheus. Later we realized that T210993: Deprecate Diamond collectors in Cloud VPS means that we also need to migrate to Prometheus for basic dashboards like T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.

We now need to redesign and build out the POC to scale to collecting at least basic instance health information for all instances in all projects for some reasonable amount of time (3+ months for sure, 1+ year ideally).

Features needed beyond POC:

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal)
 ResolvedcolewhiteT183454 Deprovision Diamond collectors no longer in use
 ResolvedMoritzMuehlenhoffT210993 Deprecate Diamond collectors in Cloud VPS
 OpenNoneT336774 Current status of cloudmetrics and its components
 ResolvedtaaviT326266 Remove the WMCS statsd/Graphite service
 OpendcaroT313444 Streamline WMCS Alerting and Paging
 ResolvedtaaviT317032 Remove Diamond?
 ResolvedtaaviT264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
 OpenNoneT194333 [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants
 ResolvedtaaviT266050 Build Prometheus service for use by all Cloud VPS projects and their instances
 Resolved• JHeddenT250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
 Resolved• JHeddenT250210 Request creation of metricsinfra VPS project
 Resolved• BstormT252260 Try out removing the nfs client probes from the node exporter on VMs
 Resolvedbd808T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
 ResolvedAndrewT128615 Get rid of Toolforge home page check from shinken
 DeclinedNoneT128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
 OpenNoneT186552 Create alerts for bastion hosts - Usage and latency
 Resolved• nskaggsT284973 Request increased quota for metricsinfra Cloud VPS project
 OpenNoneT284993 Enable self-service Prometheus configuration management for project administrators
 OpenNoneT286302 Enable external API access to metricsinfra Prometheus configuration
 ResolvedtaaviT286300 Add a database-backed backend for prometheus-configurator
 ResolvedtaaviT286299 Create initial scaffolding for Prometheus configuration automation
 OpenNoneT288052 Figure out how to deploy metricsinfra Prometheus configuration tooling
 OpenNoneT288058 Add support for more scrape targets configuration types
 ResolvedtaaviT288059 metricsinfra: Add support for static service discovery
 OpenNoneT288060 metricsinfra: Add support for Kubernetes service discovery
 OpenNoneT288061 metricsinfra: Add client TLS support for scrapes
 ResolvedtaaviT288067 metricsinfra: Add hosted Prometheus blackbox exporter
 ResolvedtaaviT347277 metricsinfra: add support dns blackbox scrapes
 OpenNoneT288166 metricsinfra: Alert routing configuration
 OpenNoneT128715 Add all Cloud VPS project administrators to the Prometheus notification group for each project
 ResolvedtaaviT285055 Allow project members/admins to ack/silence alerts of that project
 ResolvedjbondT286716 Create solution for developer account authentication for services hosted in Cloud VPS
 ResolvedtaaviT286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
 ResolvedtaaviT286335 Split metricsinfra alertmanager to separate hosts from prometheus
 ResolvedtaaviT287148 Set up IRC relay for metricsinfra alerts
 OpenNoneT287349 Deploy prometheus pushgateway to metricsinfra
 OpentaaviT288053 Add external meta-monitoring for metricsinfra
 OpenNoneT288168 metricsinfra: Build out default alert rules
 ResolvedtaaviT310799 Upgrade metricsinfra prometheus to bullseye
 ResolveddcaroT310802 Request increased quota for metricsinfra Cloud VPS project
 ResolvedAndrewT288108 Figure out how to deal with security groups when rolling out metricsinfra scraping

Event Timeline

bd808 created this task.Oct 20 2020, 5:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2020, 5:02 PM
bd808 added a parent task: T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.Oct 20 2020, 5:03 PM
bd808 added a subtask: T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.
bd808 added a parent task: T194333: [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants.Oct 20 2020, 5:05 PM
bd808 added a subtask: T186552: Create alerts for bastion hosts - Usage and latency.
bd808 added a subtask: T128715: Add all Cloud VPS project administrators to the Prometheus notification group for each project.Oct 20 2020, 5:08 PM
bd808 closed subtask T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken as Resolved.Oct 20 2020, 5:15 PM
bd808 mentioned this in T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.
bd808 added a comment.Oct 20 2020, 5:18 PM

The POC project was a bit ahead of work by the Observability folks on similar alerting for the production realm. This build out should include examining the profiles and modules that have been built for prod now to see how much of the POC can be replaced with shared setup.

bd808 triaged this task as High priority.Oct 20 2020, 5:23 PM
• Bstorm mentioned this in T186552: Create alerts for bastion hosts - Usage and latency.Oct 21 2020, 4:22 PM
• nskaggs subscribed.Oct 27 2020, 7:50 PM
bd808 mentioned this in T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.Dec 3 2020, 8:54 PM
taavi subscribed.Dec 13 2020, 6:14 PM
taavi mentioned this in T95922: Adopt service status dashboard.Feb 8 2021, 4:59 PM
• Bstorm mentioned this in T284860: Prometheus alerting support on Toolforge.Jun 14 2021, 4:22 PM
taavi added a subtask: T284973: Request increased quota for metricsinfra Cloud VPS project.Jun 15 2021, 5:32 AM
• nskaggs closed subtask T284973: Request increased quota for metricsinfra Cloud VPS project as Resolved.Jun 15 2021, 4:12 PM
taavi updated the task description. (Show Details)Jun 16 2021, 1:05 PM
taavi updated the task description. (Show Details)Jun 16 2021, 2:42 PM
taavi mentioned this in T286299: Create initial scaffolding for Prometheus configuration automation.Jul 7 2021, 5:53 PM
Stashbot added a comment.Jul 14 2021, 10:35 AM

Mentioned in SAL (#wikimedia-cloud) [2021-07-14T10:35:46Z] <majavah> undeploy old ingress T266050

taavi added a comment.Jul 14 2021, 10:38 AM
In T266050#7211735, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2021-07-14T10:35:46Z] <majavah> undeploy old ingress T266050

Wrong task, was meant to be T264221.

taavi closed subtask T286335: Split metricsinfra alertmanager to separate hosts from prometheus as Resolved.Jul 30 2021, 2:24 PM
taavi closed subtask T287148: Set up IRC relay for metricsinfra alerts as Resolved.Aug 4 2021, 3:00 PM
taavi removed a subtask: T128715: Add all Cloud VPS project administrators to the Prometheus notification group for each project.Aug 7 2021, 11:57 AM
• nnikkhoui subscribed.Sep 14 2021, 9:23 PM
taavi updated the task description. (Show Details)Oct 22 2021, 1:39 PM
dcaro subscribed.Mar 3 2022, 9:00 AM
Hydriz subscribed.Apr 28 2022, 7:19 AM
taavi mentioned this in T307655: Replacement needed for obsolete Diamond/Graphite monitoring of integration instances.May 5 2022, 7:51 AM
taavi mentioned this in T308555: Provide access to Thanos.May 18 2022, 7:36 AM
taavi updated the task description. (Show Details)Jun 16 2022, 4:06 PM
taavi closed subtask T285055: Allow project members/admins to ack/silence alerts of that project as Resolved.Jun 17 2022, 7:18 PM
taavi closed subtask T286301: Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar) as Resolved.Jul 6 2022, 9:32 AM
taavi updated the task description. (Show Details)Jul 6 2022, 11:04 AM
taavi closed subtask T310799: Upgrade metricsinfra prometheus to bullseye as Resolved.Oct 28 2022, 3:07 PM
taavi closed subtask T288108: Figure out how to deal with security groups when rolling out metricsinfra scraping as Resolved.Nov 15 2022, 7:14 AM
taavi claimed this task.Nov 15 2022, 9:02 AM
taavi updated the task description. (Show Details)
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 7:12 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
taavi closed this task as Resolved.Feb 5 2023, 12:12 PM

While there are still open feature requests as subtasks, I'm closing this task as the metricsinfra Prometheus service is now monitoring all instances and can replace Diamond.

• nskaggs awarded a token.Feb 10 2023, 9:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL