I'm creating this task to track an incident that happened earlier this week.
It triggered the alert ProjectProxyMainProxyDown, but no Phab task was created because the alertmanager-phab integration is not working for alerts in the wmcloud prometheus instance (T380901: prometheus wmcloud alerts stopped sending emails).
Incident summary:
- Systemd unit for nginx failed in proxy-04 VM
- This was caused by Puppet removing the “nameserver” IP from Nginx configuration files
- It’s a known bug already tracked in T379927: Puppet removed "nameserver" line from /etc/resolv.conf
Incident report:
https://wikitech.wikimedia.org/wiki/Incidents/2024-11-25_WMCS_proxy_nginx_failure