Ranked bots paste has been superseded by the shared doc: https://docs.google.com/spreadsheets/d/1PKfAhcc2jXl72CbF73JXTeZMTbw_RtQnYJ6YZ4Fozyk/edit?gid=0#gid=0

I don't think regexes should be automatically escaped: the user must always be in charge of deciding that (and automatically parsing what's a regex and what not in a input box could be a pain). I agree that a big bold warning about escaped/unescaped regex could be added to some fields.

In T409267#11343708, @Joe wrote:

x-provenance is supposed to only be defined in the provenance scope, so I assume you're talking about that. Correct?

haproxy configuration deployed everywhere

In T408202#11307258, @BCornwall wrote:

To be clear, this isn't a regression in recent changes, is it?

Password has been reset by @ssingh for me, thanks anyway

A not-so-refined search on Turnilo produced this paste: P84293
We can even refine it later

@Joe so, just have a better vision over some points: some of these steps must be performed manually, while other should happen in an automated fashion, like:

This has been superseded by more refined actions to exclude a broader class of "invalid" requests. As HTTP/1.0 requests are not per-se invalid we can consider this declined.

Problem is that PyBal (twisted) defaults to a HTTP1.0 client so healthchecks in eqiad|codfw will fail after this. Or we patch PyBal to support HTTP1.1 requests or we have to wait for Liberica being deployed here too (or we make an exception in HAProxy configuration for healthcheck requests but it doesn't seem a good long-term solution to me),

This has been migrated to Gitlab in the meantime

Yep, definitely, thanks for reminding

This has been reverted due to issues with load balancers checks

Done rejecting all HTTP_1.0 requests

Hi @BTullis sorry for the late answer, I think this fired correctly because being depooled the host produced no haproxykafka messages so, IMHO is the right thing to do. In this case we usually both depool and silence the affected host (if the depool lasts longer than some minutes). IIRC varnishkafka had the same behavior

We deployed a change in HAProxy logging (see T403176) to avoid sending non-utf8 encoded headers to DLQ, this *could* also affect this issue as we're now logging these messages to Kafka (through the usual HaproxyKafka pipeline).
@Antoine_Quhen could you check all is good on your side?

I'm also taking care of this with some experiments to check when actually HAProxy (or HaproxyKafka) skips these messages

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1183081 I think we can consider this as closed. New cache hosts reimaged won't have varnishkafka references (except for statsv)

In T393772#11129887, @ssingh wrote:

@Fabfur: I think this all done and the alerts have been removed as well. Confirming: can we close this ticket as resolved? If yes, please do so. Thanks!

Abandoned for T400244

Declined. Better upgrade purged to golang-1.23

Renamed to HttpHound

Closing as per T400199 and opening a new ticket dedicated only to watchdog feature

This has been addressed with the following changes:

In T400244#11044566, @CDanis wrote:

Just a note that the existing config validity tests in puppet modules/profile/files/cache/haproxy/tests have been broken by bullseye-backports no longer existing on mirrors.

This has been already fixed upstream in docker-registry.wikimedia.org/bullseye:20250723, problem is that docker (or podman, FWIW) doesn't fetch the latest tag already, so that's need to be pulled explicitly.

Note that this happened again ~2025-07-24 14:43 on cp3071, same host

Thanks a lot @Scott_French ! This weekend is fine, I'll retry on Monday!

The image used for debci building (bullseye) is still affected by the bullseye-backports issue:

Closing as we can use the HaproxyKafka component to keep track of related tasks

Adding DE team too, considering that two hosts didn't sent messages for a long time and this could be impacting on the analytics data.