Upon upgrade to version 2.4.16 HAProxy started increasing the haproxy_(frontend|backend|server)_internal_errors_total metric that also implied a increase of 5xx errors (text@esams on the screenshot):
Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
mtail::cache_haproxy: Track termination state per request | operations/puppet | production | +15 -8 |
Event Timeline
Mentioned in SAL (#wikimedia-operations) [2022-05-03T10:58:01Z] <vgutierrez> rolling downgrade of HAProxy to version 2.4.15 on text - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-03T12:57:40Z] <vgutierrez> rolling downgrade of HAProxy to version 2.4.15 on upload - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-03T14:03:36Z] <vgutierrez> upgrade haproxy to 2.4.16 on cp3050 - T307444
Change 789108 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] mtail::cache_haproxy: Track termination state per request
Change 789108 merged by Vgutierrez:
[operations/puppet@production] mtail::cache_haproxy: Track termination state per request
Mentioned in SAL (#wikimedia-operations) [2022-05-18T09:45:56Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in eqsin - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-18T11:00:37Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in drmrs - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-18T13:34:39Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in codfw - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-18T14:10:30Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in esams - T307444
Mentioned in SAL (#wikimedia-operations) [2022-05-18T15:04:31Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in eqiad - T307444
(95) cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5016].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4030,4032-4036].ulsfo.wmnet ----- OUTPUT of 'dpkg -l haproxy' ----- Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-================-============-============================================== ii haproxy 2.4.17-1~bpo10+1 amd64 fast and reliable load balancing reverse proxy
we are now running HAProxy 2.4.17