tbs: user-story 9: I want to know when the service is down
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcaro
	Dec 14 2022, 1:49 PM

Project Tags

Referenced Files

None

Subscribers

dcaro

Description

As a Toolforge service administrator (toolforge root) I want to know when the service is down

When do we know it's done?

Feature: I want to know when the service is down

Scenario: Tekton is down in toolsbeta

When tekton is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Harbor is down in toolsbeta

When harbor is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Tekton is down in tools

When tekton is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

Scenario: Harbor is down in tools

When harbor is down
Then I want to get notified by email
And I want to see the alert in the alerts dashboard (https://prometheus-alerts.wmcloud.org/)

This epic contains the following tasks:

This is a tracking task, so it will be closed once the tasks mentioned below are completed.

Related Objects
Search...

Status	Assigned	Task
Resolved	LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
Resolved	• matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
In Progress	dcaro	T194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Resolved	dcaro	T267374 [tbs.beta] Create a toolforge build service beta release
Resolved	dcaro	T380959 [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components
Resolved	dcaro	T325172 [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service
Resolved	None	T325174 [builds-builder,harbor,bulid-service,docs] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them.
Resolved	dcaro	T325166 tbs: user-story 10: I want to know how to manage the service
Resolved	dcaro	T325167 tbs: user-story 10: Create admin wiki page for the toolforge build service
Resolved	Raymond_Ndibe	T325175 tbs: user-story 11: Add a runbook for each of the service alerts.
Resolved	Raymond_Ndibe	T325160 tbs: user-story 9: I want to know when the service is down
Resolved	Raymond_Ndibe	T325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
Resolved	Raymond_Ndibe	T325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
Resolved	Raymond_Ndibe	T325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
Resolved	Raymond_Ndibe	T325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
Resolved	dcaro	T330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
Resolved	dcaro	T330757 [tbs.harbor] Harbor core being down is not noticed by simple http request