Page MenuHomePhabricator
Create Task
Maniphest T394275

[components-api] Add alerts and runbooks for basic service health
Closed, ResolvedPublic
Actions

Description

This includes:

  • Add 'up' alert

For this you have to add a new directory here https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts , similar to the other services (feel free to copy-paste and tweak after).
The alert should be something very similar to https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/blob/main/buildservice/builds-api.yaml?ref_type=heads

  • Add runbook for that alert (the previous patch will not pass the tests until you add it)

You can use https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/BuildsApiDown as inspiration, the new runbook should have '''exactly''' the new alert name as title. The focus is to help whomever gets paged by the alert a starting point to triage without having to know much context.

Related Objects
Search...

Event Timeline

dcaro created this task.May 14 2025, 8:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2025, 8:46 AM
dcaro triaged this task as High priority.May 14 2025, 8:54 AM
dcaro mentioned this in T393564: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features.May 14 2025, 9:02 AM
dcaro added a parent task: T393564: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features.
dcaro edited projects, added Toolforge (Toolforge iteration 20); removed Toolforge (Toolforge iteration 19).May 14 2025, 2:30 PM
dcaro updated the task description. (Show Details)May 22 2025, 9:23 AM
Raymond_Ndibe claimed this task.May 23 2025, 2:02 AM
Raymond_Ndibe added a project: User-Raymond_Ndibe.
CodeReviewBot added a project: Patch-For-Review.May 30 2025, 4:52 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/31

[components-service] add components api alert

Raymond_Ndibe changed the task status from Open to In Progress.May 30 2025, 5:30 PM
Raymond_Ndibe moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 20) board.
Raymond_Ndibe added a comment.Jun 10 2025, 9:40 AM

https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-cluster_datasource=P8433460076D33992&var-cluster=tools

dcaro edited projects, added Toolforge (Toolforge iteration 21); removed Toolforge (Toolforge iteration 20).Jun 11 2025, 8:58 AM
dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 21) board.
CodeReviewBot added a comment.Jun 11 2025, 3:30 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/85

prometheus: use multiproc stats

CodeReviewBot added a comment.Jun 11 2025, 3:56 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/31

[components-service] add components api alert

dcaro added a comment.Jun 12 2025, 5:03 PM

Hmm... it seems that the alerts are not being deployed in tools (they did in toolsbeta).

The alerts-deploy service seems to have some permission issues:

Jun 12 17:01:39 tools-prometheus-8 systemd[1]: Starting alerts-deploy@project-tools.service - Deploy alerts from git to Prometheus/Thanos (instance project/tools)...
░░ Subject: A start job for unit alerts-deploy@project-tools.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit alerts-deploy@project-tools.service has begun execution.
░░ 
░░ The job identifier is 19636.
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]: Traceback (most recent call last):
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:   File "/usr/local/bin/alerts-deploy", line 178, in <module>
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:     sys.exit(main())
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:              ^^^^^^
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:   File "/usr/local/bin/alerts-deploy", line 168, in main
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:     deployed_paths = deploy_rulefiles(rulefiles, deploy_dir, alerts_dir)
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:   File "/usr/local/bin/alerts-deploy", line 40, in deploy_rulefiles
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:     shutil.copy2(src_path.as_posix(), dest_path.as_posix())
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:   File "/usr/lib/python3.11/shutil.py", line 436, in copy2
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:     copyfile(src, dst, follow_symlinks=follow_symlinks)
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:   File "/usr/lib/python3.11/shutil.py", line 258, in copyfile
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:     with open(dst, 'wb') as fdst:
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]:          ^^^^^^^^^^^^^^^
Jun 12 17:01:39 tools-prometheus-8 alerts-deploy@project/tools[14227]: PermissionError: [Errno 13] Permission denied: '/srv/alerts/project-tools/kubernetes_kyverno.yaml'
Jun 12 17:01:39 tools-prometheus-8 systemd[1]: alerts-deploy@project-tools.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit alerts-deploy@project-tools.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Jun 12 17:01:39 tools-prometheus-8 systemd[1]: alerts-deploy@project-tools.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit alerts-deploy@project-tools.service has entered the 'failed' state with result 'exit-code'.
Jun 12 17:01:39 tools-prometheus-8 systemd[1]: Failed to start alerts-deploy@project-tools.service - Deploy alerts from git to Prometheus/Thanos (instance project/tools).
░░ Subject: A start job for unit alerts-deploy@project-tools.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit alerts-deploy@project-tools.service has finished with a failure.
░░ 
░░ The job identifier is 19636 and the job result is failed.
dcaro added a comment.Jun 12 2025, 5:04 PM

Just chowning seemed to do the trick:

root@tools-prometheus-8:/srv# chown -R alerts-deploy:alerts-deploy /srv/alerts
dcaro added a comment.Jun 12 2025, 5:06 PM

Now showing up on tools too \o/

image.png (554×787 px, 133 KB)

CodeReviewBot added a comment.Jun 16 2025, 1:06 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/85

prometheus: use multiproc stats

CodeReviewBot added a comment.Jun 16 2025, 1:09 PM

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/816

components-api: bump to 0.0.118-20250616130629-11e499d8

CodeReviewBot added a comment.Jun 16 2025, 2:22 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/816

components-api: bump to 0.0.118-20250616130629-11e499d8

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2025, 2:31 PM
dcaro added a comment.Jun 17 2025, 8:08 AM

All alerts are now in place in both prometheus, and the stats are correct, closing.

dcaro closed this task as Resolved.Jun 17 2025, 8:08 AM
dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 21) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits