Looks like now factgrid is working again, closing this

Yes, thanks

In T438817#12349324, @Aklapper wrote:

We can't fetch FactGrid list of crawler IPs anymore

@Fabfur: Where do we do this? Adding a project tag is welcome :)

Waiting for @ssingh confirmation and we can move on with ops group!

as fare as I can see on turnilo, after the change above I can't find occurrences of "funny" Host headers. I'll close this task but please reopen if you find other issues with that!

We've identified the issue (the Host header was captured and logged before the normalization directive). We've also now applied a fix for this, so from now on there should be no other "funny" host headers in turnilo

Can't replicate, I've just archive this page (new page, so never archived before) and (aside from occasional timeouts from web.archive.org) it worked fine. Can you paste the link or a screenshot of the error?

Hi @toddbradley , thanks for the report. We've depooled the Singapore DC some hours ago due to a scheduled maintenance, so this could be possibly related, even if your requests should be routed to another DC anyway. Apparently this is completely different from the issue we experienced yesterday globally.
Can you provide some more information, eg. are you using a VPN somehow that could route your to different countries? Is this still persists or it works fine now?

Thanks for the updates @Cyberpower678!
Could you investigate if this is still an issue?
If someone from archive.org wants to directly reach us (also mail channels works for us) and describe what they are observing on their (server) side when trying to fetch from our servers, this could speed up the process (and not bother you as relay :D )

Thanks for the update! Can you still confirm the archive.org is remained the same and/or no changes has been made regard to the network from where these requests comes from?

I've noticed that archiving a new page from Commons, Web Archive completes the task successfully: it fetches the page and the image (and thumbnails) as expected.
But when visiting the relative page on Web Archive the image thumbnail isn't showed (from browser console, it returns a 500 for that).
This supports also the lack of errors on our side when looking at the archive bot UA that retrieves content.

from a rapid look at our logs, we're not currently blocking archive.org bot and trying to save a couple of pages from different wiki (enwiki, itwiki) works fine (even if possibly a bit slow, so it could reach some timeout at some point).

That's very strange because we should've already fixed this (at least for port 443) with T392880

I think this can be marked as resolve, thanks @Ladsgroup !

I'm going to close this ticket, please reopen if needed

Changes has been merged into puppet and should become effective soon. Marking the ticket as "resolved", please reopen if needed

Changes has been merged into puppet and should become effective soon. Marking the ticket as "resolved", please reopen if needed

Marking this as resolved, please reopen if needed

Change has been merged to add user to the fr-tech-devs, should be effective soon

@Jniren-ctr merging the change today and this should be completed

Hi @Bmueller , @MSantos , the change has been deployed, user should have access soon to the services, closing this ticket but feel free to reopen if something isn't working as expected!

I think this can be closed

In T427068#12034446, @JAllemandou wrote:

I have checked the kafka streams, the new field is present in both webrequest_frontend_text and webrequest_sampled.
The ball is now in the DE team side :)
Thank you @Fabfur for making this happen !

In T419825#12050927, @MoritzMuehlenhoff wrote:

Are the results of the comparison between aws-lc and openssl available somewhere?

All hosts with haproxy-awlc reimaged with standard haproxy 3.2 version

Checking with @Ahoelzl and @JAllemandou on where to go from there.
Also I think some documentation like https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Webrequest should be upgraded

In T427068#12033025, @Volans wrote:

Do you need anything added to the superset dashboard for this?

I'll start writing some HAProxy configuration for this. In theory we can deploy a new version of HaproxyKafka before adding the actual header in HAProxy, and HaproxyKafka will populate with the default (-) in this case, but I prefer to have the header already in place (HaproxyKafka will ignore it until we upgrade).

So IIUC in this case we have 2 possibilities:

I always assumed the ones in /etc/haproxy/errors/ but they are unused in our current configuration. If we want to actually use them to mimic eventual varnish error pages we should re-introduce the configuration. Is it worth?

Thanks!

In T423570#11902575, @mxn wrote:

In T423570#11902251, @Fabfur wrote:

If not possible at the moment I suggest to modify the configuration of QuickInstantCommons to increase descriptionCacheExpiry and apiMetadataExpiry values or introduce thumbCacheExpiry setting to increase the cache expiration time. This way you should not hit the limits so quickly...

Should that be apiThumbCacheExpiry, which defaults to 1 day?

I suggest to going the route of not using the thumbnails from upload.w.o at all, if not really needed, to avoid incurring in rate-limiting that we're currently setting up to protect our media infrastructure.

I think this tag can be used both for statistics and to categorize tickets to work on given the amount of reports that could be received in the near future about this. My thoughts on this are that such tasks can be opened both by end-user and by us.

The number of this kind of errors should be lower now, as we've tuned the rule to accomodate for higher ratelimiting

In T352744#11840575, @ssingh wrote:

In T352744#11840551, @Ladsgroup wrote:

In T352744#9413282, @MoritzMuehlenhoff wrote:

In T352744#9413140, @jhathaway wrote:

wolfssl is packaged in Debian, so that may be a possible option longer term, https://tracker.debian.org/pkg/wolfssl.

wolfssl isn't fully supported in Debian, though: It's basically only included for some applications stuck with GPLv2 and where specific concerns with OpenSSL exist (for some upstream which don't want to rely on the "system library clause" of the GPL), it's not something we can comfortably use hapoxy with IMO.

Hate to be that guy but many large places now have switched to boringssl. Do you think it'd be worthwhile to look at that?

I am not the person working on this, @Fabfur is so defer to his opinion, but the reason we didn't consider BoringSSL is because even per them, their API is unstable and so applications depending on it will have a hard time keeping up.

Thanks, this should've been closed long long time ago...

In T422030#11794907, @Vgutierrez wrote:

It looks like the root cause is MEDIUM: log/session: handle embryonic session log within sess_log(). A change introduced in HAProxy 3.1 as part of the work done to introduce the log profiles feature.

From the commit message:

Also, thanks to this change, log-format-sd will now be taken into account
for legacy embryonic session logging.

All these new BADREQ rows are actually TLS handshake errors that under 3.0 were being discarded by haproxykafka cause the log message wasn't a valid RFC5424 message.

Related patch

The patch has been applied to all impacted hosts, thanks @JAllemandou for spotting this!

This is most probably due to a deprecation in haproxy configuration directives https://www.haproxy.com/blog/reviewing-every-new-feature-in-haproxy-3-1#deprecation, especially