[docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Nov 27 2024, 10:09 AM

Description

There seems to be missing docs on how to operate the cluster, for example how to restart a given core component in case of suspected misbehavior.

Example of why this is important is the ticket T380832: [jobs-api] crashing in which an operator responded to an incident did not know how to restart the jobs-api.

We could:

create docs
- builds-service https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Build_Service
- envvars-service https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Envvars_Service
- jobs-service (review https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Jobs_Service)
crate alerts
- builds-service https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/21
- envvars-service https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/22
- jobs-service
  - jobs-api https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/20
  - jobs-emailer https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10
create automation
have some training for team members

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	envvars-service: add alerts	repos/cloud/toolforge/alerts!22	dcaro	add_envvars_alert	main
	builds-api: add up alert	repos/cloud/toolforge/alerts!21	dcaro	add_builds_api_alert	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	dcaro	T380959 [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components
Resolved	dcaro	T325172 [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service
Resolved	None	T325174 [builds-builder,harbor,bulid-service,docs] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them.
Resolved	dcaro	T325166 tbs: user-story 10: I want to know how to manage the service
Resolved	dcaro	T325167 tbs: user-story 10: Create admin wiki page for the toolforge build service
Resolved	Raymond_Ndibe	T325175 tbs: user-story 11: Add a runbook for each of the service alerts.
Resolved	Raymond_Ndibe	T325160 tbs: user-story 9: I want to know when the service is down
Resolved	Raymond_Ndibe	T325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
Resolved	Raymond_Ndibe	T325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
Resolved	Raymond_Ndibe	T325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
Resolved	Raymond_Ndibe	T325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
Resolved	dcaro	T330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
Resolved	dcaro	T330757 [tbs.harbor] Harbor core being down is not noticed by simple http request
Resolved	dcaro	T320284 [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components
Resolved	dcaro	T379924 [jobs-emailer] http requests are blocked by the loops

Event Timeline

• aborrero created this task.Nov 27 2024, 10:09 AM

dcaro added subtasks: T325172: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service, T325166: tbs: user-story 10: I want to know how to manage the service.Nov 27 2024, 10:32 AM

dcaro added a subtask: T320284: [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.

dcaro updated the task description. (Show Details)Nov 27 2024, 10:36 AM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/21

builds-api: add up alert

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/22

envvars-service: add alerts

dcaro updated the task description. (Show Details)Nov 27 2024, 11:29 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/21

builds-api: add up alert

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/22

envvars-service: add alerts

dcaro renamed this task from toolforge: create docs on how to operate the cluster and core components to [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.Nov 27 2024, 2:21 PM

dcaro triaged this task as High priority.

dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge.

dcaro updated the task description. (Show Details)Nov 27 2024, 2:30 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 2:31 PM