Page MenuHomePhabricator

Upgrade HAProxy on cp nodes to 2.6.x LTS
Open, In Progress, MediumPublic

Description

2.6 is the next LTS version of HAProxy. it's been released on 2022-05-31 and on it's current 2.6.6 it can be useful in several aspects:

  • OpenSSL 3.0 support
  • HTTP/3 over QUIC support (it can't be enabled right away due to lack of QUIC support in current OpenSSL versions)
  • tcp-request connection set-var
  • set-var-fmt action added (uses the logging message string format to set variable values)
  • The new global directive close-spread-time lets you close idle connections gradually over a period of time, rather than all at once.

We are currently running HAProxy 2.6.6 in the following hosts:

  • deployment-cache-text07
  • deployment-cache-upload07
  • cp1075
  • cp2027
  • cp3050
  • cp3051
  • cp3052
  • cp5007

Event Timeline

Vgutierrez triaged this task as Medium priority.Oct 27 2022, 9:01 AM
Vgutierrez updated the task description. (Show Details)

Change 850416 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] aptrepo: Add thirdparty/haproxy26

https://gerrit.wikimedia.org/r/850416

Change 850417 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Allow choosing between HAProxy 2.4 and 2.6

https://gerrit.wikimedia.org/r/850417

Change 850420 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Switch to HAProxy 2.6 on concurrency tracking instances

https://gerrit.wikimedia.org/r/850420

Vgutierrez changed the task status from Open to In Progress.Oct 28 2022, 9:40 AM
Vgutierrez moved this task from Triage to In Progress on the Traffic board.

Change 850416 merged by Vgutierrez:

[operations/puppet@production] aptrepo: Add thirdparty/haproxy26

https://gerrit.wikimedia.org/r/850416

Change 850417 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Allow choosing between HAProxy 2.4 and 2.6

https://gerrit.wikimedia.org/r/850417

Mentioned in SAL (#wikimedia-releng) [2022-11-02T09:56:57Z] <vgutierrez> update to HAProxy 2.6.6 in deployment-cache-(text|upload)07 - T321775

Current config isn't valid for HAProxy 2.6.6:

vgutierrez@deployment-cache-text07:~$ sudo -i haproxy -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c
[NOTICE]   (15054) : haproxy version is 2.6.6-1~bpo10+1
[NOTICE]   (15054) : path to executable is /usr/sbin/haproxy
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:3] : nbproc is not supported any more since HAProxy 2.5. Threads will automatically be used on multi-processor machines if available.
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:54] : error detected while parsing ACL 'missing_xwd' : matching method must be specified first (using '-m') when using a sample fetch of this type ('var').
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:74] : error detected while parsing an 'http-response del-header' condition : no such ACL : 'missing_xwd'.
[ALERT]    (15054) : config : Error(s) found in configuration file : /etc/haproxy/conf.d/tls.cfg

Change 852141 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy: Produce valid configs for HAProxy 2.6.x

https://gerrit.wikimedia.org/r/852141

Change 852141 merged by Vgutierrez:

[operations/puppet@production] haproxy: Produce valid configs for HAProxy 2.6.x

https://gerrit.wikimedia.org/r/852141

Mentioned in SAL (#wikimedia-operations) [2022-11-02T10:57:20Z] <vgutierrez> depool cp1075, cp2027 and cp3050 prior to HAProxy 2.6 upgrade - T321775

Change 850420 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Switch to HAProxy 2.6 on concurrency tracking instances

https://gerrit.wikimedia.org/r/850420

Mentioned in SAL (#wikimedia-operations) [2022-11-02T11:13:47Z] <vgutierrez> pool cp1075, cp2027 and cp3050 running HAProxy 2.6.6 - T321775

Change 852211 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Update cp305[12] to HAProxy 2.6

https://gerrit.wikimedia.org/r/852211

Change 852211 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Update cp305[12] to HAProxy 2.6

https://gerrit.wikimedia.org/r/852211