Add alerting to Uptime Monitoring of public wikibase.cloud services
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Tarrow
	Dec 7 2021, 12:53 PM

Description

If we wish to be able to act to ensure that our service it up it is valuable to both monitor what fraction of the time it is down as well as alert us in the event this happens.

These should alert to the existing monitoring email.

Some wbstack.com alerts that were setup in terraform can be found at https://github.com/wbstack/deploy/blob/main/tf/monitoring_alert_policy.tf

Useful links:

Defining uptime checks https://cloud.google.com/monitoring/uptime-checks/introduction
Notification channels https://cloud.google.com/monitoring/support/notification-options
Alerting https://cloud.google.com/monitoring/alerts

A/C:

Monitoring exists to check the uptime of all our public facing services
Alerting exists to notify developers in a real time way to wb-cloud-monitoring@wikimedia.de of one of these public facing services has been unavailable for 10 mins.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Declined		None	T299508 [EPIC] Platform status and performance monitoring implemented for Wikibase.cloud
		Resolved		Evelien_WMDE	T297198 Add alerting to Uptime Monitoring of public wikibase.cloud services

Event Timeline

Tarrow created this task.Dec 7 2021, 12:53 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2021, 12:53 PM

Addshore moved this task from Backlog (incoming) to Ready to Pick Up on the Wikibase Cloud board.Jan 7 2022, 6:41 PM

I added basic public facing uptime checks for most services in https://github.com/wmde/wbaas-deploy/pull/136
the API is an exception here, and we probably need to add a route that can be used as a health check.

Also we still want to add alerting for checks etc, so leaving this in ready to pick up!

Addshore renamed this task from Add Monitoring to Wikibase.dev to Add Uptime Monitoring to public Wikibase.dev services.Jan 7 2022, 7:23 PM

Addshore updated the task description. (Show Details)

Addshore updated the task description. (Show Details)Jan 7 2022, 7:25 PM

This first PR, https://github.com/wmde/wbaas-deploy/pull/136, has been deployed now as a starting point (as it was approved and in the repo review queue)

These initial checks can be found under uptime https://console.cloud.google.com/monitoring/uptime?referrer=search&project=wikibase-cloud

The checks are still rolling out, but they each have at least 1 success now

• Samantha_Alipio_WMDE moved this task from Ready to Pick Up to Backlog (incoming) on the Wikibase Cloud board.Jan 25 2022, 3:26 PM

• toan added a parent task: T299508: [EPIC] Platform status and performance monitoring implemented for Wikibase.cloud.Feb 23 2022, 10:19 AM

Tarrow renamed this task from Add Uptime Monitoring to public Wikibase.dev services to Add Uptime Monitoring and Alerting to public wikibase.cloud services.Feb 23 2022, 8:51 PM

Tarrow updated the task description. (Show Details)

Tarrow moved this task from Backlog (incoming) to Ready to Pick Up on the Wikibase Cloud board.

• Samantha_Alipio_WMDE moved this task from Ready to Pick Up to Backlog (incoming) on the Wikibase Cloud board.Mar 9 2022, 10:23 AM

Tarrow moved this task from Backlog (incoming) to Product prioritized backlog on the Wikibase Cloud board.Jun 21 2022, 2:41 PM

Tarrow moved this task from Product prioritized backlog to Engineering prioritised backlog on the Wikibase Cloud board.Jun 23 2022, 9:40 PM

Evelien_WMDE moved this task from Engineering prioritised backlog to Product prioritized backlog on the Wikibase Cloud board.Jul 12 2022, 3:04 PM