08:05:47 <dcaro> hmm, it seems all the requests that fail go through the same tools-k8s-gateway node, that might be the issue
08:06:56 <dcaro> can you open a task? we'll follow up in a bit
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | taavi | T431946 2026-07-11 Incident - Toolforge haproxies flapping | |||
| Open | None | T431967 Intermittent timeouts on Phab-ban tool |
Event Timeline
Comment Actions
The logs of me doing requests that triggered that comment:
root@tools-k8s-haproxy-8:~# grep k8s-ingress-http/tools-k8s-gateway-2 /var/log/haproxy/haproxy.log.1 | grep phab-ban | wc
13 299 6139
root@tools-k8s-haproxy-8:~# grep k8s-ingress-http/tools-k8s-gateway-2 /var/log/haproxy/haproxy.log.1 | grep phab-ban | grep -Pe ' (503|400) ' | wc
12 276 5657The requests through the other gateways all went ok.
Comment Actions
Overall, the rate is a bit higher than the other two gateways:
root@tools-k8s-haproxy-8:~# for i in 1 2 3; do errors=$(grep k8s-ingress-http/tools-k8s-gateway-$i /var/log/haproxy/haproxy.log | grep -Pe ' (503|400) ' | wc -l); all=$(grep k8s-ingress-http/tools-k8s-gateway-$i /var/log/haproxy/haproxy.log | wc -l); percent=$(((100 * $errors) / $all)); echo "gateway-$i: $percent % errors"; done gateway-1: 7 % errors gateway-2: 12 % errors gateway-3: 7 % errors