root@deployment-cache-upload08:~# haproxy -c -f /etc/haproxy/haproxy.cfg 
[NOTICE]   (1149872) : haproxy version is 2.8.18-1~bpo11+1
[NOTICE]   (1149872) : path to executable is /usr/sbin/haproxy
[ALERT]    (1149872) : config : parsing [/etc/haproxy/haproxy.cfg:11] : unknown keyword 'warn-blocked-traffic-after' in 'global' section
[ALERT]    (1149872) : config : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT]    (1149872) : config : Fatal errors found in configuration.
root@deployment-cache-upload08:~#

This is happening since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1295007, where @CDanis added warn-blocked-traffic-after 500ms. This parameter doesn't seem to be present in haproxy 2.8 on beta.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T401839 Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie
		Open		None	T428052 Beta cluster haproxy does not support `warn-blocked-traffic-after` keyword

Event Timeline

Urbanecm_WMF created this task.Wed, Jun 3, 12:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Jun 3, 12:38 PM

Pushed this on beta as a stopgap:

root@deployment-puppetserver-1:/srv/git/operations/puppet# git diff HEAD~ HEAD
diff --git a/modules/profile/templates/cache/haproxy.cfg.erb b/modules/profile/templates/cache/haproxy.cfg.erb
index 37f6508a94..c7b394a111 100644
--- a/modules/profile/templates/cache/haproxy.cfg.erb
+++ b/modules/profile/templates/cache/haproxy.cfg.erb
@@ -17,7 +17,8 @@ global
     hard-stop-after 1m
     set-dumpable
     nbthread <%= num_threads %>
-    warn-blocked-traffic-after 500ms
+    # commented out, as it is not recognised in beta (T428052)
+    # warn-blocked-traffic-after 500ms
 <%
     htsets = @facts['numa']['device_to_htset'][@numa_iface]
     cpus = htsets.flatten.join(' ')
root@deployment-puppetserver-1:/srv/git/operations/puppet#

Sgs subscribed.Wed, Jun 3, 12:52 PM

Urbanecm_WMF added projects: SRE, Traffic.Wed, Jun 3, 12:56 PM

In T428052#11981214, @Urbanecm_WMF wrote:

Pushed this on beta as a stopgap:

Thanks for the workaround @Urbanecm_WMF.

When possible we like to track the stack of Beta ~~hacks~~workarounds in Gerrit using hashtag:beta-cherry-picked. For [BETA HACK] patches specifically we have been using the WIP status to indicate in Gerrit that the patch is not a thing we expect to be merged. Hopefully this one won't live long enough to make it worthwhile to setup all that tracking, but because you are the sort of awesome person who is likely to help in this area again in the future I thought it would be worth pointing out the best practices.

The Beta Cluster cache nodes are Debian Bullseye running HAProxy version 2.8.18-1~bpo11+1 2025/12/26. It looks like the prod CDN edge is using Debian Trixie and HAProxy 3.2.15-1~bpo13+1.

• bd808 added a parent task: T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie.Wed, Jun 3, 4:11 PM

In T428052#11982183, @bd808 wrote:

The Beta Cluster cache nodes are Debian Bullseye running HAProxy version 2.8.18-1~bpo11+1 2025/12/26. It looks like the prod CDN edge is using Debian Trixie and HAProxy 3.2.15-1~bpo13+1.

See also: T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie

Yes that is correct, for the CDN side. In this particular case, we can put it under one of our usual guards.

• bd808 moved this task from To Triage to Puppet errors on the Beta-Cluster-Infrastructure board.Wed, Jun 3, 11:30 PM

• bd808 merged a task: T427819: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep.Thu, Jun 4, 12:10 AM

• bd808 merged a task: T427813: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep.

• bd808 added a subscriber: wmcs-alerts.

Beta cluster haproxy does not support `warn-blocked-traffic-after` keywordOpen, HighPublicActions

Description

Related ObjectsSearch...

Event Timeline

Beta cluster haproxy does not support `warn-blocked-traffic-after` keyword
Open, HighPublic
Actions

Related Objects
Search...