Maniphest T306495

🟦️ Add alerting for failed logical backups
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Deniz_WMDE
	Apr 20 2022, 9:01 AM

Tags

Referenced Files

None

Subscribers

Description

As developers we want to get an alert for the logical backup cronjob failing.

possible solutions for alert:

log metric of cronjob logs
container/restart_count

ACs

alert is triggered if backup cronjob didn't succeed
alert is send to the existing notification group

relevant links:

https://cloud.google.com/monitoring/alerts/types-of-conditions

Event Timeline

Deniz_WMDE created this task.Apr 20 2022, 9:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2022, 9:01 AM

WMDE-leszek moved this task from Launch Sprint 7 (2022) to Launch Migration Kanban (2022) on the Wikibase Cloud board.Apr 20 2022, 1:00 PM

WMDE-leszek edited projects, added Wikibase Cloud (Launch Migration Kanban (2022)); removed Wikibase Cloud (Launch Sprint 7 (2022)).

WMDE-leszek triaged this task as High priority.Apr 20 2022, 1:27 PM

• toan renamed this task from Add alerting for missing logical backups to Add alerting for failed logical backups.Apr 20 2022, 1:44 PM

Deniz_WMDE updated the task description. (Show Details)Apr 20 2022, 1:50 PM

Deniz_WMDE updated the task description. (Show Details)Apr 20 2022, 1:56 PM

Rosalie_WMDE claimed this task.Apr 27 2022, 11:56 AM

Rosalie_WMDE moved this task from Backlog to Doing on the Wikibase Cloud (Launch Migration Kanban (2022)) board.

Rosalie_WMDE removed Rosalie_WMDE as the assignee of this task.Apr 29 2022, 1:06 PM

Rosalie_WMDE subscribed.

Rosalie_WMDE claimed this task.May 3 2022, 8:39 AM

https://github.com/wmde/wbaas-deploy/pull/285

Rosalie_WMDE removed Rosalie_WMDE as the assignee of this task.May 12 2022, 8:01 AM

Rosalie_WMDE moved this task from Doing to Review on the Wikibase Cloud (Launch Migration Kanban (2022)) board.

• toan claimed this task.May 12 2022, 8:34 AM

• toan removed • toan as the assignee of this task.May 12 2022, 10:30 AM

• toan subscribed.

• toan moved this task from Review to To Release on the Wikibase Cloud (Launch Migration Kanban (2022)) board.May 12 2022, 11:55 AM

https://github.com/wmde/wbaas-deploy/pull/291
https://github.com/wmde/wbaas-deploy/pull/297

Rosalie_WMDE moved this task from To Release to Deploy To Production on the Wikibase Cloud (Launch Migration Kanban (2022)) board.May 12 2022, 1:57 PM

I suggest we wait about 2 days before deploying this to production ( Just to monitor it on staging. ) because there was an alert fired about 30 minutes after this was deployed to staging. I did some small investigation and I am not sure if the query started counting from the last backup before the alert was deployed.

WMDE-leszek renamed this task from Add alerting for failed logical backups to 🟦️ Add alerting for failed logical backups.May 17 2022, 7:18 PM

https://github.com/wmde/wbaas-backup/pull/18

https://github.com/wmde/wbaas-deploy/pull/326

Rosalie_WMDE moved this task from Deploy To Production to Done on the Wikibase Cloud (Launch Migration Kanban (2022)) board.May 19 2022, 10:18 AM

Tarrow closed this task as Resolved.Jun 23 2022, 9:14 PM

Tarrow claimed this task.