tbs: user-story 9: I want to know when the service is down
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	dcaro
	Dec 14 2022, 1:49 PM

Description

As a Toolforge service administrator (toolforge root) I want to know when the service is down

When do we know it's done?

Feature: I want to know when the service is down

Scenario: Tekton is down in toolsbeta

When tekton is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Harbor is down in toolsbeta

When harbor is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Tekton is down in tools

When tekton is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Harbor is down in tools

When harbor is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

This epic contains the following tasks:

This is a tracking task, so it will be closed once the tasks mentioned below are completed.

Related Objects
Search...

Status	Assigned	Task
Stalled	LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
Stalled	matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Open	None	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
Open	None	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
Open	dcaro	T194332 [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs
In Progress	dcaro	T267374 [tbs.beta] Create a toolforge build service beta release
Open	None	T325172 tbs: user-story 11: I want to know how to debug the service
Open	None	T325174 tbs: user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them.
Open	None	T325166 tbs: user-story 10: I want to know how to manage the service
Open	komla	T325167 tbs: user-story 10: Create admin wiki page for the toolforge build service
Open	None	T325175 tbs: user-story 11: Add a runbook for each of the service alerts.
Open	None	T325160 tbs: user-story 9: I want to know when the service is down
Open	None	T325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
Open	None	T325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
Open	None	T325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
Open	None	T325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
Resolved	dcaro	T330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
Resolved	dcaro	T330757 [tbs.harbor] Harbor core being down is not noticed by simple http request