Thanks, this should've been closed long long time ago...

In T422030#11794907, @Vgutierrez wrote:

It looks like the root cause is MEDIUM: log/session: handle embryonic session log within sess_log(). A change introduced in HAProxy 3.1 as part of the work done to introduce the log profiles feature.

From the commit message:

Also, thanks to this change, log-format-sd will now be taken into account
for legacy embryonic session logging.

All these new BADREQ rows are actually TLS handshake errors that under 3.0 were being discarded by haproxykafka cause the log message wasn't a valid RFC5424 message.

Related patch

[[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1266301 | The patch ]]has been applied to all impacted hosts, thanks @JAllemandou for spotting this!

This is most probably due to a deprecation in haproxy configuration directives https://www.haproxy.com/blog/reviewing-every-new-feature-in-haproxy-3-1#deprecation, especially

This could be related to upgrade to HAProxy 3.2 (T421402) that started on the drmrs datacenter, we'll investigate if the sequence-id is now generated differently

Had to partially revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1261484 because configuration gets applied before upgrading HAProxy package (with the cookbook) so it would break the HAProxy reload and hence puppet run. This patch will be reverted at the very end to enable that configuration.

Procedure from the traffic perspective should be roughly

Hi @daniel I can confirm I see the x_is_browser, trusted_req and rl_class fields in webrequest (text), good job!

The patch has been merged and will be propagated shortly to all cache hosts. This means that the headers X-WMF-Ratelimit-Class and X-Trusted-Request will soon be present in X-Analytics

Thanks to the @JAllemandou summary, I've wrote down some simple notes to understand the amount of work and the missing pieces, please have a look and eventually comment/fix:

Good for me! Just let me know as soon as you have the "official" header name so I can make the relevant changes in HAProxy logging and HaproxyKafka configuration

@daniel it could be helpful to have some (also fake) example of values for X-WMF-Ratelimit-Class headers, to do some optimization in the pipeline, do you have some ?

In T401832#11603084, @BCornwall wrote:

After some IRC chatter the proposed path is:

Bullseye 2.8 → Bullseye 3.0 → Trixie 3.0 → Trixie 3.2 (or 3.3, depending on TLS library needs)

This is because 3.2 is not available for Bullseye: If we're to upgrade while still running Bullseye it has to be 3.0.

@Xqt we're rolling out a change that should lift the current ratelimiting and impact Pywikibot too, could you please check in ~30 minutes if you still see the same amount of errors?
Thanks

We're now allowing this new type of contact information in User-Agent string, this change should be propagated shortly. Please notify us on this ticket if the situation remains unchanged

In T414173#11507588, @JAnD wrote:

In T414173#11507562, @Fabfur wrote:

I think this is better, if you want to add an email as contact you can do it right after the URL, separating the two with a ;.

xx.wikipedia.org/wiki/User:Botname can be easily obtainet from current user configuration. There is nowhere operators email and this framework is used by hundrets of users. This bug/change is breaking most bots

In T414173#11507526, @Xqt wrote:

@Fabfur: Currently we have this UA:
version (wikipedia:de; User:Xqtest) Pywikibot/11.0.0.dev10 (g20136) requests/2.32.5 Python/3.13.0.final.0
Would this be appropriate:
version (https://de.wikipedia:org/User:Xqtest) Pywikibot/11.0.0.dev10 (g20136) requests/2.32.5 Python/3.13.0.final.0

In T414173#11507457, @Xqt wrote:

In T414173#11507270, @Joe wrote:

This user agent is not compliat with our user-agent policy:

https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy

which requires you to provide either an email or a specific url for your bot.

If you change that, it should allow you a higher rate-limit.

Pywikibot's UA wasn't changed for 12 years but I'll try to fix it. Where can I check whether the new format string matches?

The entry has been added with https://gerrit.wikimedia.org/r/c/operations/dns/+/1220380 and should be propagated shortly

Update: we've tested this on cp7009 but apparently this isn't setting the TOS as expected. I've rolled it back with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1218788 to try again tomorrow morning and check if this could be an issue on the haproxy side (configuration, capabilities should be ok).

Ranked bots paste has been superseded by the shared doc: https://docs.google.com/spreadsheets/d/1PKfAhcc2jXl72CbF73JXTeZMTbw_RtQnYJ6YZ4Fozyk/edit?gid=0#gid=0

I don't think regexes should be automatically escaped: the user must always be in charge of deciding that (and automatically parsing what's a regex and what not in a input box could be a pain). I agree that a big bold warning about escaped/unescaped regex could be added to some fields.

In T409267#11343708, @Joe wrote:

x-provenance is supposed to only be defined in the provenance scope, so I assume you're talking about that. Correct?

haproxy configuration deployed everywhere

In T408202#11307258, @BCornwall wrote:

To be clear, this isn't a regression in recent changes, is it?

Password has been reset by @ssingh for me, thanks anyway

A not-so-refined search on Turnilo produced this paste: P84293
We can even refine it later

@Joe so, just have a better vision over some points: some of these steps must be performed manually, while other should happen in an automated fashion, like:

This has been superseded by more refined actions to exclude a broader class of "invalid" requests. As HTTP/1.0 requests are not per-se invalid we can consider this declined.

Problem is that PyBal (twisted) defaults to a HTTP1.0 client so healthchecks in eqiad|codfw will fail after this. Or we patch PyBal to support HTTP1.1 requests or we have to wait for Liberica being deployed here too (or we make an exception in HAProxy configuration for healthcheck requests but it doesn't seem a good long-term solution to me),

This has been migrated to Gitlab in the meantime

Yep, definitely, thanks for reminding

This has been reverted due to issues with load balancers checks

Done rejecting all HTTP_1.0 requests

Hi @BTullis sorry for the late answer, I think this fired correctly because being depooled the host produced no haproxykafka messages so, IMHO is the right thing to do. In this case we usually both depool and silence the affected host (if the depool lasts longer than some minutes). IIRC varnishkafka had the same behavior

We deployed a change in HAProxy logging (see T403176) to avoid sending non-utf8 encoded headers to DLQ, this *could* also affect this issue as we're now logging these messages to Kafka (through the usual HaproxyKafka pipeline).
@Antoine_Quhen could you check all is good on your side?

I'm also taking care of this with some experiments to check when actually HAProxy (or HaproxyKafka) skips these messages

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1183081 I think we can consider this as closed. New cache hosts reimaged won't have varnishkafka references (except for statsv)

In T393772#11129887, @ssingh wrote:

@Fabfur: I think this all done and the alerts have been removed as well. Confirming: can we close this ticket as resolved? If yes, please do so. Thanks!

Abandoned for T400244

Declined. Better upgrade purged to golang-1.23