Page MenuHomePhabricator
Create Task
Maniphest T320284

[jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components
Closed, ResolvedPublic
Actions

Description

This should include:

  • Retrieving metrics on prometheus side (if there's anything missing)
  • Add alerts for "down" events - with runbooks
  • Add a basic grafana board with the "up/down" metric to add as 'dashboard' to the alerts
  • jobs-api
    • gather stats
    • add alert
  • jobs-emailer
    • gather stats
    • add alerts

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge: add jobs-emailer stats gatheringoperations/puppetproduction+6 -0
P:toolforge::prometheus: scrape jobs-apioperations/puppetproduction+6 -0
Configure prometheus flask exportercloud/toolforge/jobs-framework-apimain+35 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
jobs-emailer: add basic alertsrepos/cloud/toolforge/alerts!28dcaroadd_jobs_emailer_basicmain
jobs-emailer: bump to 0.0.52-20250219170643-a14ae54drepos/cloud/toolforge/toolforge-deploy!670ghostbump_jobs-emailermain
jobs-api: add alerts for it being downrepos/cloud/toolforge/alerts!20dcaroadd_jobs_apimain
webserver: add a minimal metrics endpointrepos/cloud/toolforge/jobs-emailer!7dcaroadd_prometheus_statsmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

taavi created this task.Oct 7 2022, 7:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2022, 7:57 PM
gerritbot added a comment.Oct 7 2022, 7:58 PM

Change 840225 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

gerritbot added a project: Patch-For-Review.Oct 7 2022, 7:58 PM
gerritbot added a comment.Oct 10 2022, 8:36 AM

Change 840225 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

Stashbot added a comment.Oct 10 2022, 8:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:41:57Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

Stashbot added a comment.Oct 10 2022, 8:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:44:51Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

gerritbot added a comment.Oct 10 2022, 8:46 AM

Change 841033 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

gerritbot added a comment.Oct 10 2022, 9:13 AM

Change 841033 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2022, 9:30 AM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:37 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
• aborrero removed a parent task: T285944: Toolforge: beta phase for the new jobs framework.Jan 24 2023, 4:41 PM
dcaro renamed this task from Prometheus monitoring toolforge-jobs server side components to [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.Mar 11 2024, 2:32 PM
dcaro triaged this task as High priority.
dcaro edited projects, added Toolforge; removed Toolforge Jobs framework.
dcaro updated the task description. (Show Details)
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
dcaro updated the task description. (Show Details)Nov 13 2024, 3:35 PM
dcaro claimed this task.Nov 14 2024, 1:53 PM
dcaro moved this task from Ready to be worked on to Toolforge iteration 16 on the Toolforge board.
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge.
dcaro changed the task status from Open to In Progress.Nov 14 2024, 1:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 16) board.
CodeReviewBot added a project: Patch-For-Review.Nov 14 2024, 1:57 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7

webserver: add a minimal metrics endpoint

• aborrero added a subtask: T379924: [jobs-emailer] http requests are blocked by the loops.Nov 15 2024, 9:01 AM
CodeReviewBot added a comment.Nov 18 2024, 9:13 AM

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7

webserver: add a minimal metrics endpoint

Maintenance_bot removed a project: Patch-For-Review.Nov 18 2024, 9:30 AM
dcaro moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 16) board.Nov 19 2024, 10:37 AM
dcaro changed the status of subtask T379924: [jobs-emailer] http requests are blocked by the loops from Open to In Progress.
• aborrero mentioned this in T380832: [jobs-api] crashing.Nov 26 2024, 12:01 PM
dcaro updated the task description. (Show Details)Nov 27 2024, 8:36 AM
CodeReviewBot added a project: Patch-For-Review.Nov 27 2024, 8:42 AM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/20

jobs-api: add alerts for it being down

dcaro added a parent task: T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.Nov 27 2024, 10:32 AM
CodeReviewBot added a comment.Nov 27 2024, 2:27 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/20

jobs-api: add alerts for it being down

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 2:32 PM
dcaro edited projects, added Toolforge (Toolforge iteration 17); removed Toolforge (Toolforge iteration 16).Jan 8 2025, 12:48 PM
dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 17) board.
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 17) board.Jan 21 2025, 2:14 PM
CodeReviewBot added a comment.Feb 19 2025, 5:06 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10

add prometheus stats

CodeReviewBot added a project: Patch-For-Review.Feb 19 2025, 5:09 PM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/670

jobs-emailer: bump to 0.0.52-20250219170643-a14ae54d

Stashbot added a comment.Feb 19 2025, 5:22 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-02-19T17:22:11Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer (T320284)

Stashbot added a comment.Feb 19 2025, 5:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-02-19T17:30:40Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer (T320284)

Stashbot added a comment.Feb 20 2025, 1:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-02-20T13:18:13Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer (T320284)

Stashbot added a comment.Feb 20 2025, 1:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-02-20T13:26:35Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer (T320284)

CodeReviewBot added a comment.Feb 20 2025, 1:27 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/670

jobs-emailer: bump to 0.0.52-20250219170643-a14ae54d

dcaro closed subtask T379924: [jobs-emailer] http requests are blocked by the loops as Resolved.Feb 20 2025, 1:29 PM
gerritbot added a comment.Feb 20 2025, 1:36 PM

Change #1121364 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge: add jobs-emailer stats gathering

https://gerrit.wikimedia.org/r/1121364

dcaro moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 17) board.Feb 20 2025, 2:13 PM
gerritbot added a comment.Feb 20 2025, 2:24 PM

Change #1121364 merged by David Caro:

[operations/puppet@production] toolforge: add jobs-emailer stats gathering

https://gerrit.wikimedia.org/r/1121364

dcaro updated the task description. (Show Details)Feb 20 2025, 3:31 PM
dcaro added a comment.Feb 20 2025, 3:39 PM

Added them to the toolforge overview dashboard:

image.png (394×1 px, 37 KB)

dcaro closed this task as Resolved.Feb 20 2025, 3:39 PM
dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 17) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits