Page MenuHomePhabricator
Create Task
Maniphest T320284

[jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components
Open, In Progress, HighPublic
Actions

Description

This should include:

  • Retrieving metrics on prometheus side (if there's anything missing)
  • Add alerts for "down" events - with runbooks
  • Add a basic grafana board with the "up/down" metric to add as 'dashboard' to the alerts
  • jobs-api (done)
  • jobs-emailer

Details

SubjectRepoBranchLines +/-
P:toolforge::prometheus: scrape jobs-apioperations/puppetproduction+6 -0
Configure prometheus flask exportercloud/toolforge/jobs-framework-apimain+35 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi created this task.Oct 7 2022, 7:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2022, 7:57 PM
gerritbot added a comment.Oct 7 2022, 7:58 PM

Change 840225 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

gerritbot added a project: Patch-For-Review.Oct 7 2022, 7:58 PM
gerritbot added a comment.Oct 10 2022, 8:36 AM

Change 840225 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

Stashbot added a comment.Oct 10 2022, 8:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:41:57Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

Stashbot added a comment.Oct 10 2022, 8:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:44:51Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

gerritbot added a comment.Oct 10 2022, 8:46 AM

Change 841033 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

gerritbot added a comment.Oct 10 2022, 9:13 AM

Change 841033 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2022, 9:30 AM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:37 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
aborrero removed a parent task: T285944: Toolforge: beta phase for the new jobs framework.Jan 24 2023, 4:41 PM
dcaro renamed this task from Prometheus monitoring toolforge-jobs server side components to [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.Mar 11 2024, 2:32 PM
dcaro triaged this task as High priority.
dcaro edited projects, added Toolforge; removed Toolforge Jobs framework.
dcaro updated the task description. (Show Details)
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
dcaro updated the task description. (Show Details)Wed, Nov 13, 3:35 PM
dcaro claimed this task.Thu, Nov 14, 1:53 PM
dcaro moved this task from Ready to be worked on to Toolforge iteration 16 on the Toolforge board.
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge.
dcaro changed the task status from Open to In Progress.Thu, Nov 14, 1:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 16) board.
CodeReviewBot added a project: Patch-For-Review.Thu, Nov 14, 1:57 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7

webserver: add a minimal metrics endpoint

aborrero added a subtask: T379924: [jobs-emailer] http requests are blocked by the loops.Fri, Nov 15, 9:01 AM
CodeReviewBot added a comment.Mon, Nov 18, 9:13 AM

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7

webserver: add a minimal metrics endpoint

Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 18, 9:30 AM
dcaro moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 16) board.Tue, Nov 19, 10:37 AM
dcaro changed the status of subtask T379924: [jobs-emailer] http requests are blocked by the loops from Open to In Progress.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits