Change Details

The Toolforge HTTP ingress setup had some problems today.

The Toolforge HTTP ingress setup had some problems today. Alerts flapping for haproxy backends going down, ex: {F58608488} The trouble started today at ~1:00 am UTC: https://tools-prometheus.wmflabs.org/tools/graph?g0.expr=haproxy_server_up%20%3C%201&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=12h {F58608402} I was able to reproduce the issue by just refreshing any tool page, or doing a manual curl like: ``` root@tools-k8s-haproxy-5:~# time curl -v http://tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud:30002 ``` And it timing out sometimes (but responding quick others). There did not seem to be any issues with memory or cpu limits. I restarted the pods from the ingress-nginx-gen2 deployment and things improved, I can't manually reproduce anymore and haproxy retries and errors went down: https://usercontent.irccloud-cdn.com/file/5pNj6dqz/image.png https://usercontent.irccloud-cdn.com/file/5QJjU7QI/image.png But alerts are still flapping.

The Toolforge HTTP ingress setup had some problems today. Alerts flapping for haproxy backends going down, ex: {F58608488} The trouble started today at ~1:00 am UTC: https://tools-prometheus.wmflabs.org/tools/graph?g0.expr=haproxy_server_up%20%3C%201&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=12h {F58608402} I was able to reproduce the issue by just refreshing any tool page, or doing a manual curl like: ``` root@tools-k8s-haproxy-5:~# time curl -v http://tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud:30002 ``` And it timing out sometimes (but responding quick others). There did not seem to be any issues with memory or cpu limits. I restarted the pods from the ingress-nginx-gen2 deployment and things improved, I can't manually reproduce anymore and haproxy retries and errors went down: https://usercontent.irccloud-cdn.com/file/5pNj6dqz/image.png https://usercontent.irccloud-cdn.com/file/5QJjU7QI/image.png But alerts are still flapping.