Page MenuHomePhabricator

HAProxy 2.4.16 shows internal errors on text cluster
Closed, ResolvedPublic

Description

Upon upgrade to version 2.4.16 HAProxy started increasing the haproxy_(frontend|backend|server)_internal_errors_total metric that also implied a increase of 5xx errors (text@esams on the screenshot):

image.png (878×1 px, 138 KB)

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2022-05-03T10:58:01Z] <vgutierrez> rolling downgrade of HAProxy to version 2.4.15 on text - T307444

Mentioned in SAL (#wikimedia-operations) [2022-05-03T12:57:40Z] <vgutierrez> rolling downgrade of HAProxy to version 2.4.15 on upload - T307444

Vgutierrez lowered the priority of this task from High to Medium.May 3 2022, 12:59 PM

Lowering the priority as after downgrading text we aren't experiencing more issues

Mentioned in SAL (#wikimedia-operations) [2022-05-03T14:03:36Z] <vgutierrez> upgrade haproxy to 2.4.16 on cp3050 - T307444

Change 789108 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] mtail::cache_haproxy: Track termination state per request

https://gerrit.wikimedia.org/r/789108

Change 789108 merged by Vgutierrez:

[operations/puppet@production] mtail::cache_haproxy: Track termination state per request

https://gerrit.wikimedia.org/r/789108

Mentioned in SAL (#wikimedia-operations) [2022-05-18T09:45:56Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in eqsin - T307444

Mentioned in SAL (#wikimedia-operations) [2022-05-18T11:00:37Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in drmrs - T307444

Mentioned in SAL (#wikimedia-operations) [2022-05-18T13:34:39Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in codfw - T307444

Mentioned in SAL (#wikimedia-operations) [2022-05-18T14:10:30Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in esams - T307444

Mentioned in SAL (#wikimedia-operations) [2022-05-18T15:04:31Z] <vgutierrez> rolling upgrade to HAProxy 2.4.17 in eqiad - T307444

(95) cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5016].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4030,4032-4036].ulsfo.wmnet        
----- OUTPUT of 'dpkg -l haproxy' -----                                                                                                                                           
Desired=Unknown/Install/Remove/Purge/Hold                                                                                                                                         
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend                                                                                                    
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version          Architecture Description
+++-==============-================-============-==============================================
ii  haproxy        2.4.17-1~bpo10+1 amd64        fast and reliable load balancing reverse proxy

we are now running HAProxy 2.4.17