Page MenuHomePhabricator
Create Task
Maniphest T320284

[jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components
Open, HighPublic
Actions

Description

This should include:

  • Retrieving metrics on prometheus side (if there's anything missing)
  • Add alerts for "down" events - with runbooks
  • Add a basic grafana board with the "up/down" metric to add as 'dashboard' to the alerts

Details

SubjectRepoBranchLines +/-
P:toolforge::prometheus: scrape jobs-apioperations/puppetproduction+6 -0
Configure prometheus flask exportercloud/toolforge/jobs-framework-apimain+35 -0
Customize query in gerrit

Related Objects

Event Timeline

taavi created this task.Oct 7 2022, 7:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2022, 7:57 PM
gerritbot added a comment.Oct 7 2022, 7:58 PM

Change 840225 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

gerritbot added a project: Patch-For-Review.Oct 7 2022, 7:58 PM
gerritbot added a comment.Oct 10 2022, 8:36 AM

Change 840225 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] Configure prometheus flask exporter

https://gerrit.wikimedia.org/r/840225

Stashbot added a comment.Oct 10 2022, 8:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:41:57Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

Stashbot added a comment.Oct 10 2022, 8:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-10T08:44:51Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko

gerritbot added a comment.Oct 10 2022, 8:46 AM

Change 841033 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

gerritbot added a comment.Oct 10 2022, 9:13 AM

Change 841033 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] P:toolforge::prometheus: scrape jobs-api

https://gerrit.wikimedia.org/r/841033

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2022, 9:30 AM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:37 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
aborrero removed a parent task: T285944: Toolforge: beta phase for the new jobs framework.Jan 24 2023, 4:41 PM
dcaro renamed this task from Prometheus monitoring toolforge-jobs server side components to [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.Mar 11 2024, 2:32 PM
dcaro triaged this task as High priority.
dcaro edited projects, added Toolforge; removed Toolforge Jobs framework.
dcaro updated the task description. (Show Details)
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL