Maniphest T325175

tbs: user-story 11: Add a runbook for each of the service alerts.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcaro
	Dec 14 2022, 3:06 PM

Tags

Referenced Files

None

Subscribers

Description

For each of the alerts created here:

T325160: tbs: user-story 9: I want to know when the service is down

Add a runbook (see https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks) with step-to-step debugging.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
		Resolved		matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
		Resolved		Legoktm	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
		Resolved		Legoktm	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
		In Progress		dcaro	T194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
		Resolved		dcaro	T267374 [tbs.beta] Create a toolforge build service beta release
		Resolved		dcaro	T380959 [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components
		Resolved		dcaro	T325172 [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service
		Resolved		Raymond_Ndibe	T325175 tbs: user-story 11: Add a runbook for each of the service alerts.
		Resolved		Raymond_Ndibe	T325160 tbs: user-story 9: I want to know when the service is down
		Resolved		Raymond_Ndibe	T325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
		Resolved		Raymond_Ndibe	T325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
		Resolved		Raymond_Ndibe	T325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
		Resolved		Raymond_Ndibe	T325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
		Resolved		dcaro	T330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
		Resolved		dcaro	T330757 [tbs.harbor] Harbor core being down is not noticed by simple http request

Event Timeline

dcaro created this task.Dec 14 2022, 3:06 PM

dcaro triaged this task as High priority.

dcaro added a project: Toolforge Build Service.

dcaro added a subtask: T325160: tbs: user-story 9: I want to know when the service is down.Dec 14 2022, 3:07 PM

KHernandez-WMF mentioned this in T325172: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service.Dec 19 2022, 11:52 PM

Raymond_Ndibe added a project: User-Raymond_Ndibe.Jan 17 2023, 10:27 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:25 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

dcaro raised the priority of this task from High to Needs Triage.Mar 6 2023, 3:02 PM

https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/TektonDown
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown

Raymond_Ndibe closed this task as Resolved.May 9 2023, 11:40 PM

Raymond_Ndibe claimed this task.

Raymond_Ndibe edited projects, added Toolforge Build Service (Iteration 14); removed Toolforge Build Service.

Raymond_Ndibe moved this task from Next Up to Done on the Toolforge Build Service (Iteration 14) board.

Raymond_Ndibe changed the status of subtask T325160: tbs: user-story 9: I want to know when the service is down from Open to In Progress.

Raymond_Ndibe closed subtask T325160: tbs: user-story 9: I want to know when the service is down as Resolved.May 10 2023, 5:09 PM