Change Details

**This requires the statistic generated in the subtask**. As of writing this task, creating the alert can only be done directly in the DB, more info here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS Essentially, we have a cloud vps project, metricsinfra, where we have a setup with prometheus(alertmanager), specifically, there's a couple hosts: metricsinfra-controller-1.metricsinfra.eqiad1.wikimedia.cloud metricsinfra-controller-2.metricsinfra.eqiad1.wikimedia.cloud That generate the alerts for prometheus from a DB, that is hosted in trove. You have to login into that DB (you can find the credentials and host in the controller hosts config, /etc/prometheus-manager/config.yaml). There you have the `prometheusconfig` database, with the table `alerts`, that you have to update with the alerts that you want to add, an example row: ``` *************************** 1. row *************************** id: 1 project_id: 12 name: GridQueueProblem expr: sge_queueproblems{project="toolsbeta",state=~".*(e|E).*"} duration: 30m severity: warn annotations: {"summary": "Grid queue {{ $labels.queue }}@{{ $labels.host }} is in state {{ $labels.state }}", "runbook": "https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem"} ``` The column `expr` is the prometheus expression that you want to monitor, you can find out, check and test them here: https://prometheus.wmflabs.org/ Another place you can use to find the expression to use is: https://grafana-rw.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace?orgId=1&var-cluster=prometheus-toolsbeta&var-namespace=image-build&forceLogin&search=open Inspecting the graphs there and the datasources you will be able to see which prometheus instance and which expression are the ones that give you the data you want. About the alert itself, it should have also an annotation called 'service' with the value 'toolforge,build_service'.

This can be achieved by reusing the blackbox probe, that is used in puppet with something like: ``` prometheus::blackbox::check::http { $static_domain: port => 80, # this should always exist path => '/admin/fingerprints/', ip_families => ['ip4'], prometheus_instance => 'tools', team => 'wmcs', severity => 'warning', } ``` Note that this should be attached to the harbor profile, and the path should be changed to something harbor will reply 2xx on when up.

**This requires the statistic generated in the subtask**. As of writing this task, creating the alert can only be done directly in the DB, more info here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS Essentially, we have a cloud vps project, metricsinfra, where we have a setup with prometheus(alertmanager), specifically, there's a couple hosts: metricsinfra-controller-1.metricsinfra.eqiad1.wikimedia.cloud metricsinfra-controller-2.metricsinfra.eqiad1.wikimedia.cloud That generate the alerts for prometheus from a DB, that is hosted in trove. You have to login into that DB (you can find the credentials and host in the controller hosts config, /etc/prometheus-manager/config.yaml). There you have the `prometheusconfig` database, with the table `alerts`, that you have to update with the alerts that you want to addThis can be achieved by reusing the blackbox probe, an example row:that is used in puppet with something like: ``` *************************** 1. row *************************** prometheus::blackbox::check::http { $static_domain: id: 1port => 80, project_id: 12 # this should always exist name: GridQueueProblem path => '/admin/fingerprints/', expr: sge_queueproblems{project="toolsbeta",state=~".*(e|E).*"} ip_families => ['ip4'], duration: 30m prometheus_instance => 'tools', severity: warn team => 'wmcs', annotations: {"summary": "Grid queue {{ $labels.queue }}@{{ $labels.host }} is in state {{ $labels.state }}", "runbook": "https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem"} severity => 'warning', ``` The column `expr` is the prometheus expression that you want to monitor, you can find out, check and test them here: https://prometheus.wmflabs.org/ Another place you can use to find the expression to use is: https://grafana-rw.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace?orgId=1&var-cluster=prometheus-toolsbeta&var-namespace=image-build&forceLogin&search=open Inspecting the graphs there and the datasources you will be able to see which prometheus instance and which expression are the ones that give you the data you want. About the alert itself, it should have also an annotation called 'service' with the value 'toolforge,build_service'. } ``` Note that this should be attached to the harbor profile, and the path should be changed to something harbor will reply 2xx on when up.