Page MenuHomePhabricator
Create Task
Maniphest T266050

Build Prometheus service for use by all Cloud VPS projects and their instances
Closed, ResolvedPublic
Actions

Description

In T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken we worked on a proof of concept project to design a replacement for monitoring a subset of projects using Shinken with Prometheus. Later we realized that T210993: Deprecate Diamond collectors in Cloud VPS means that we also need to migrate to Prometheus for basic dashboards like T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.

We now need to redesign and build out the POC to scale to collecting at least basic instance health information for all instances in all projects for some reasonable amount of time (3+ months for sure, 1+ year ideally).

Features needed beyond POC:

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal)
 ResolvedcolewhiteT183454 Deprovision Diamond collectors no longer in use
 ResolvedMoritzMuehlenhoffT210993 Deprecate Diamond collectors in Cloud VPS
 ResolvedtaaviT336774 Current status of cloudmetrics and its components
 ResolvedtaaviT326266 Remove the WMCS statsd/Graphite service
 OpendcaroT313444 Streamline WMCS Alerting and Paging
 ResolvedtaaviT317032 Remove Diamond?
 ResolvedtaaviT264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
 OpenNoneT194333 [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants
 ResolvedtaaviT266050 Build Prometheus service for use by all Cloud VPS projects and their instances
 Resolved• JHeddenT250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
 Resolved• JHeddenT250210 Request creation of metricsinfra VPS project
 Resolved• BstormT252260 Try out removing the nfs client probes from the node exporter on VMs
 Resolvedbd808T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
 ResolvedAndrewT128615 Get rid of Toolforge home page check from shinken
 DeclinedNoneT128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
 Resolved• nskaggsT284973 Request increased quota for metricsinfra Cloud VPS project
 ResolvedtaaviT285055 Allow project members/admins to ack/silence alerts of that project
 ResolvedjbondT286716 Create solution for developer account authentication for services hosted in Cloud VPS
 ResolvedtaaviT286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
 ResolvedtaaviT286335 Split metricsinfra alertmanager to separate hosts from prometheus
 ResolvedtaaviT287148 Set up IRC relay for metricsinfra alerts
 ResolvedtaaviT310799 Upgrade metricsinfra prometheus to bullseye
 ResolveddcaroT310802 Request increased quota for metricsinfra Cloud VPS project
 ResolvedAndrewT288108 Figure out how to deal with security groups when rolling out metricsinfra scraping

Event Timeline

bd808 created this task.Oct 20 2020, 5:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2020, 5:02 PM
bd808 added a parent task: T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.Oct 20 2020, 5:03 PM
bd808 added a subtask: T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.
bd808 added a parent task: T194333: [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants.Oct 20 2020, 5:05 PM
bd808 added a subtask: T186552: Create alerts for bastion hosts - Usage and latency.
bd808 added a subtask: T128715: Add all Cloud VPS project administrators to the Prometheus notification group for each project.Oct 20 2020, 5:08 PM
bd808 closed subtask T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken as Resolved.Oct 20 2020, 5:15 PM
bd808 mentioned this in T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken.
bd808 added a comment.Oct 20 2020, 5:18 PM

The POC project was a bit ahead of work by the Observability folks on similar alerting for the production realm. This build out should include examining the profiles and modules that have been built for prod now to see how much of the POC can be replaced with shared setup.

bd808 triaged this task as High priority.Oct 20 2020, 5:23 PM
• Bstorm mentioned this in T186552: Create alerts for bastion hosts - Usage and latency.Oct 21 2020, 4:22 PM
• nskaggs subscribed.Oct 27 2020, 7:50 PM
bd808 mentioned this in T264920: Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus.Dec 3 2020, 8:54 PM
taavi subscribed.Dec 13 2020, 6:14 PM
taavi mentioned this in T95922: Adopt service status dashboard.Feb 8 2021, 4:59 PM
• Bstorm mentioned this in T284860: Prometheus alerting support on Toolforge.Jun 14 2021, 4:22 PM
taavi added a subtask: T284973: Request increased quota for metricsinfra Cloud VPS project.Jun 15 2021, 5:32 AM
• nskaggs closed subtask T284973: Request increased quota for metricsinfra Cloud VPS project as Resolved.Jun 15 2021, 4:12 PM
taavi updated the task description. (Show Details)Jun 16 2021, 1:05 PM
taavi updated the task description. (Show Details)Jun 16 2021, 2:42 PM
taavi mentioned this in T286299: Create initial scaffolding for Prometheus configuration automation.Jul 7 2021, 5:53 PM
Stashbot added a comment.Jul 14 2021, 10:35 AM

Mentioned in SAL (#wikimedia-cloud) [2021-07-14T10:35:46Z] <majavah> undeploy old ingress T266050

taavi added a comment.Jul 14 2021, 10:38 AM
In T266050#7211735, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2021-07-14T10:35:46Z] <majavah> undeploy old ingress T266050

Wrong task, was meant to be T264221.

taavi closed subtask T286335: Split metricsinfra alertmanager to separate hosts from prometheus as Resolved.Jul 30 2021, 2:24 PM
taavi closed subtask T287148: Set up IRC relay for metricsinfra alerts as Resolved.Aug 4 2021, 3:00 PM
taavi removed a subtask: T128715: Add all Cloud VPS project administrators to the Prometheus notification group for each project.Aug 7 2021, 11:57 AM
• nnikkhoui subscribed.Sep 14 2021, 9:23 PM
taavi updated the task description. (Show Details)Oct 22 2021, 1:39 PM
dcaro subscribed.Mar 3 2022, 9:00 AM
Hydriz subscribed.Apr 28 2022, 7:19 AM
taavi mentioned this in T307655: Replacement needed for obsolete Diamond/Graphite monitoring of integration instances.May 5 2022, 7:51 AM
taavi mentioned this in T308555: Provide access to Thanos.May 18 2022, 7:36 AM
taavi updated the task description. (Show Details)Jun 16 2022, 4:06 PM
taavi closed subtask T285055: Allow project members/admins to ack/silence alerts of that project as Resolved.Jun 17 2022, 7:18 PM
taavi closed subtask T286301: Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar) as Resolved.Jul 6 2022, 9:32 AM
taavi updated the task description. (Show Details)Jul 6 2022, 11:04 AM
taavi closed subtask T310799: Upgrade metricsinfra prometheus to bullseye as Resolved.Oct 28 2022, 3:07 PM
taavi closed subtask T288108: Figure out how to deal with security groups when rolling out metricsinfra scraping as Resolved.Nov 15 2022, 7:14 AM
taavi claimed this task.Nov 15 2022, 9:02 AM
taavi updated the task description. (Show Details)
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 7:12 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
taavi closed this task as Resolved.Feb 5 2023, 12:12 PM

While there are still open feature requests as subtasks, I'm closing this task as the metricsinfra Prometheus service is now monitoring all instances and can replace Diamond.

• nskaggs awarded a token.Feb 10 2023, 9:42 PM
aborrero reopened this task as Open.Sep 4 2024, 11:02 AM
aborrero added a project: Epic.
aborrero subscribed.

I'm boldly reopen to keep this "epic" task as the entry point for all the other subtasks.

taavi closed this task as Resolved.Sep 28 2024, 1:17 PM

No, please create new tasks to track future work. This task was for the initial buildout that is now complete.

taavi removed subtasks: T288168: metricsinfra: Build out default alert rules, T288053: Add external meta-monitoring for metricsinfra, T287349: Deploy prometheus pushgateway to metricsinfra, T284993: Enable self-service Prometheus configuration management for project administrators, T186552: Create alerts for bastion hosts - Usage and latency.Sep 28 2024, 1:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits