Filing this for production errors and pages today
7:29:04 PM <icinga-wm> PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - appservers-https_443: Servers mw2313.codfw.wmnet, mw2409.codfw.wmnet, mw2438.codfw.wmnet, mw2331.codfw.wmnet, mw2392.codfw.wmnet, mw2414.codfw.wmnet, mw2312.codfw.wmnet, mw2375.codfw.wmnet, mw2338.codfw.wmnet, mw2325.codfw.wmnet, mw2393.codfw.wmnet, mw2314.codfw.wmnet, mw2386.codfw.wmnet, mw2408.codfw.wmnet, mw2387.codfw.wmnet, mw2269.codfw.wmnet, mw
500 error spike
https://grafana.wikimedia.org/d/O_OXJyTVk/home-w-wiki-status?orgId=1&from=1707353807985&to=1707357407985