Add external meta-monitoring for metricsinfra
Open, MediumPublic
Actions

Assigned To

None

Authored By

	taavi
	Aug 4 2021, 10:15 AM

Description

We can't rely on metricsinfra to alert us if metricsinfra itself breaks. When it's mature enough to be usable by the general Cloud VPS community, we'll want to add external monitoring to it so we're aware of any issues inside it.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
P:wmcs::metricsinfra: add meta monitoring app skeleton	operations/puppet	production	+89 -0
P:wmcs::metriscinfra: haproxy: add route for meta monitor service	operations/puppet	production	+17 -0
team-wmcs: metricsinfra: page when alertmanager is unreachable	operations/alerts	master	+51 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• aborrero	T296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
Resolved		• aborrero	T297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
Resolved		taavi	T341060 openstack eqiad1: introduce cloud-private and cloudlb
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Open	Goal	None	T328502 Move WMCS off of Icinga and introduce alertmanager
Open		None	T345983 Remove Icinga checks for Cloud VPS projects (not: infrastructure)
Open		None	T313030 [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones
Open		None	T347148 Determine how to monitor services in cloud-private / cloudlb
Open		None	T288053 Add external meta-monitoring for metricsinfra

Event Timeline

taavi created this task.Aug 4 2021, 10:15 AM

taavi triaged this task as Lowest priority.

taavi added a parent task: T313030: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones.Jul 14 2022, 2:09 PM

taavi mentioned this in T347148: Determine how to monitor services in cloud-private / cloudlb.Sep 22 2023, 1:05 PM

taavi raised the priority of this task from Lowest to Medium.Sep 25 2023, 10:47 AM

taavi added a parent task: T347148: Determine how to monitor services in cloud-private / cloudlb.

taavi mentioned this in T347694: monitoring: find out how we could have been paged for outage "Multiple CloudVPS instances lost their IPs".Sep 29 2023, 1:23 PM

dcaro added a project: Sustainability (Incident Followup).Sep 29 2023, 2:56 PM

I am going to be working on this. The general plan is that there'll be a VM in metricsinfra that hosts a toolschecker-style web service to do a bunch of checks, and the cloud Prometheus instance can monitor that (via the outbound web proxy).

Change 966804 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::metricsinfra: add meta monitoring app skeleton

https://gerrit.wikimedia.org/r/966804

Change 966805 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::metriscinfra: haproxy: add route for meta monitor service

https://gerrit.wikimedia.org/r/966805

BCornwall added a project: SRE-OnFire.Oct 23 2023, 3:09 PM

Change 982788 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/alerts@master] team-wmcs: metricsinfra: page when alertmanager is unreachable

https://gerrit.wikimedia.org/r/982788

Change 982788 merged by jenkins-bot:

[operations/alerts@master] team-wmcs: metricsinfra: page when alertmanager is unreachable