It's currently severity: page, but I think we should only page if both proxy VMs are down. If only one is down, Keepalived should make sure that the remaining one is still serving the traffic.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
The runbook part is already tracked as T361873: ProjectProxyMainProxyDown should have a runbook page.
Comment Actions
There are two options for paging only when both are down:
- Add a new scrape job for the VIP, and alert on that (more reliable, but needs more config)
- Add a new alert that alerts when all {project="project-proxy", job="main-nginx-https"} targets are down (easier to configure, but would miss certain keepalived misconfigurations)
Comment Actions
MariaDB [prometheusconfig]> select * from alerts where id in (15, 30)\G
*************************** 1. row ***************************
id: 15
project_id: 15
name: MainProxyInstanceDown
expr: probe_success{project="project-proxy", job="main-nginx-https"} == 0
duration: 2m
severity: critical
annotations: {"summary": "Proxy on {{ $labels.instance }} is down", "runbook": "https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown"}
*************************** 2. row ***************************
id: 30
project_id: 15
name: MainProxyDown
expr: probe_success{project="project-proxy",job="main-proxy-https-vip"} == 0
duration: 2m
severity: page
annotations: {"summary": "Proxy service address is unreachable", "runbook": "https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown"}
2 rows in set (0.001 sec)Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2025-04-28T13:29:09Z] <taavi> split ProjectProxyDown alert into separate ones for a single instance and the main VIP being down T381107