Upgrade HAProxy on cp nodes to 2.6.x LTS
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Vgutierrez
	Oct 27 2022, 9:01 AM

Description

2.6 is the next LTS version of HAProxy. it's been released on 2022-05-31 and on it's current 2.6.6 it can be useful in several aspects:

OpenSSL 3.0 support
HTTP/3 over QUIC support (it can't be enabled right away due to lack of QUIC support in current OpenSSL versions)
tcp-request connection set-var
set-var-fmt action added (uses the logging message string format to set variable values)
The new global directive close-spread-time lets you close idle connections gradually over a period of time, rather than all at once.

We are currently running HAProxy 2.6.6 in the following hosts:

deployment-cache-text07
deployment-cache-upload07
cp1075
cp2027
cp3050
cp3051
cp3052
cp5007

Details

Subject	Repo	Branch	Lines +/-
cache::haproxy: Update to 2.6.8-2 globally	operations/puppet	production	+1 -9
cache::haproxy: Update to 2.6.8 in eqsin	operations/puppet	production	+2 -0
cache::haproxy: Update to 2.6.8 in ulsfo	operations/puppet	production	+2 -0
esitest: compat with haproxy >= 2.5	operations/puppet	production	+2 -0
cache::haproxy: Update cp305[12] to HAProxy 2.6	operations/puppet	production	+2 -0
cache::haproxy: Switch to HAProxy 2.6 on concurrency tracking instances	operations/puppet	production	+2 -0
haproxy: Produce valid configs for HAProxy 2.6.x	operations/puppet	production	+11 -2
cache::haproxy: Allow choosing between HAProxy 2.4 and 2.6	operations/puppet	production	+10 -9
aptrepo: Add thirdparty/haproxy26	operations/puppet	production	+22 -0

Customize query in gerrit

Event Timeline

Vgutierrez created this task.Oct 27 2022, 9:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2022, 9:01 AM

Vgutierrez triaged this task as Medium priority.Oct 27 2022, 9:01 AM

Vgutierrez updated the task description. (Show Details)

Maintenance_bot added a project: SRE.Oct 27 2022, 9:29 AM

CDanis updated the task description. (Show Details)Oct 27 2022, 1:20 PM

KOfori subscribed.Oct 27 2022, 6:03 PM

Change 850416 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] aptrepo: Add thirdparty/haproxy26

https://gerrit.wikimedia.org/r/850416

gerritbot added a project: Patch-For-Review.Oct 28 2022, 9:03 AM

Change 850417 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Allow choosing between HAProxy 2.4 and 2.6

https://gerrit.wikimedia.org/r/850417

Change 850420 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Switch to HAProxy 2.6 on concurrency tracking instances

https://gerrit.wikimedia.org/r/850420

Vgutierrez changed the task status from Open to In Progress.Oct 28 2022, 9:40 AM

Vgutierrez moved this task from Backlog to Traffic team actively servicing on the Traffic board.

Change 850416 merged by Vgutierrez:

[operations/puppet@production] aptrepo: Add thirdparty/haproxy26

https://gerrit.wikimedia.org/r/850416

Change 850417 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Allow choosing between HAProxy 2.4 and 2.6

https://gerrit.wikimedia.org/r/850417

Mentioned in SAL (#wikimedia-releng) [2022-11-02T09:56:57Z] <vgutierrez> update to HAProxy 2.6.6 in deployment-cache-(text|upload)07 - T321775

Current config isn't valid for HAProxy 2.6.6:

vgutierrez@deployment-cache-text07:~$ sudo -i haproxy -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c
[NOTICE]   (15054) : haproxy version is 2.6.6-1~bpo10+1
[NOTICE]   (15054) : path to executable is /usr/sbin/haproxy
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:3] : nbproc is not supported any more since HAProxy 2.5. Threads will automatically be used on multi-processor machines if available.
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:54] : error detected while parsing ACL 'missing_xwd' : matching method must be specified first (using '-m') when using a sample fetch of this type ('var').
[ALERT]    (15054) : config : parsing [/etc/haproxy/conf.d/tls.cfg:74] : error detected while parsing an 'http-response del-header' condition : no such ACL : 'missing_xwd'.
[ALERT]    (15054) : config : Error(s) found in configuration file : /etc/haproxy/conf.d/tls.cfg

Change 852141 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy: Produce valid configs for HAProxy 2.6.x

https://gerrit.wikimedia.org/r/852141

Change 852141 merged by Vgutierrez:

[operations/puppet@production] haproxy: Produce valid configs for HAProxy 2.6.x

https://gerrit.wikimedia.org/r/852141

Mentioned in SAL (#wikimedia-operations) [2022-11-02T10:57:20Z] <vgutierrez> depool cp1075, cp2027 and cp3050 prior to HAProxy 2.6 upgrade - T321775

Change 850420 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Switch to HAProxy 2.6 on concurrency tracking instances

https://gerrit.wikimedia.org/r/850420

Mentioned in SAL (#wikimedia-operations) [2022-11-02T11:13:47Z] <vgutierrez> pool cp1075, cp2027 and cp3050 running HAProxy 2.6.6 - T321775

Maintenance_bot removed a project: Patch-For-Review.Nov 2 2022, 11:31 AM

Change 852211 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Update cp305[12] to HAProxy 2.6

https://gerrit.wikimedia.org/r/852211

gerritbot added a project: Patch-For-Review.Nov 2 2022, 2:52 PM

Change 852211 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Update cp305[12] to HAProxy 2.6

https://gerrit.wikimedia.org/r/852211

Vgutierrez updated the task description. (Show Details)Nov 2 2022, 3:01 PM

Change 884052 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] esitest: compat with haproxy >= 2.5

https://gerrit.wikimedia.org/r/884052

Change 884052 abandoned by BBlack:

[operations/puppet@production] esitest: compat with haproxy >= 2.5

Reason:

https://gerrit.wikimedia.org/r/884052

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2023, 9:31 PM

2.6.6 has been running as expected since the experiment started, next week we plan to upgrade the whole CDN

In T321775#8596987, @Vgutierrez wrote:

2.6.6 has been running as expected since the experiment started, next week we plan to upgrade the whole CDN

We should upgrade to 2.6.8, though? Otherwise we regress on the recently-fixed-for-2.2 CVE-2023-0056

That already happened along the 2.4.21 upgrade

In T321775#8597953, @Vgutierrez wrote:

That already happened along the 2.4.21 upgrade

Yes, that's my point. We fixed CVE-2023-0056 with the upgrade to 2.4.21, so moving to 2.6.6 would revert to a build of haproxy without that fixed. But I'm seeing 2.6.8 on the pilot installs now, so all good.

yeah.. I meant that along upgrading the 2.4 hosts to 2.4.21 I also updated the 2.6 ones to 2.6.8 :)

Change 888632 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Update to 2.6.8 in ulsfo

https://gerrit.wikimedia.org/r/888632

gerritbot added a project: Patch-For-Review.Feb 13 2023, 8:45 AM

Change 888632 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Update to 2.6.8 in ulsfo

https://gerrit.wikimedia.org/r/888632

Mentioned in SAL (#wikimedia-operations) [2023-02-13T09:44:40Z] <vgutierrez> rolling upgrade to HAProxy 2.6.8 in ulsfo - T321775

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2023, 10:31 AM

Change 889053 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Update to 2.6.8 in eqsin