Hi, sorry, let's take a step back.

I disagree with the idea that we can solve a page rather than acknowledge it. It means that at that point, we cant unacknowledge a page, to let it escalate; nor we can receive a clear notification on our pager when the issue is resolved.

(Analysis below written by Giuseppe 1.0 on moderate effort) :P

I don't see why we should exclude robots.txt from our robot policy. By not allowing to fetch it and indicating the UA policy as soon as someone wants to browse us, we give them the maximum opportunity to follow our rules rather than not.

Looking at responses from zotero, I see a lot of 501 not implemented responses. Not sure if this has any relevance.

I am pretty sure what's going on is that some upstream service is slowing up responses to us intentionally. I've roll-restarted both citoid and zotero to exclude any inner state issue. The kubernetes cluster is healthy, every metric I can look at for the pods is healthy; the latency can only be explained with either a slowness of our url-downloader service (but that would be *very* strange, and also happening not just for citoid), or with an upstream service slowness.

Also: eqiad is showing a 45% failure of citations, which might be due to it getting more traffic, as it's the current active datacenter.

The problem is also limited to eqiad, showing that the problem is probably not systemic - further excluding url-downloader as the culprit.

So: I don't think this is an incident with user-facing consequences. It lines up with an increase in requests and corresponding 404s. Unless we have evidence otherwise, I would focus on finding out which requests are resulting in 404s by looking at access logs for citoid.

The increase in latency seems related to an increase in 404 results. Has anyone investigated that? It's very possible that the issue is related to someone requesting resources that can't be found.

Since 15:38 there's been an increase in timeouts for wiki requests

The pattern pywikibot uses for user-agents is nto only supported, but also unit-tested.

This is clearly some bot (or set of bots) requesting that page again and again. The requests are almost all coming from ISPs notorious for hosting residential proxies. This is just a campaign to increase pageview counts for a single page that we didn't properly detect.

I will add a couple of points here:

FWIW, I think this introduces both a discrepancy with the logic we adopt at the edge, and a matter of inequality: bots who control a large enough IP space would have a much larger limit than the bot of a community member, which is typically coming from a single source IP.

If you want to expose the HTTP API for users, that would happen via the component the linked artifact service would expose. I don't understand what the issue is here.

The way I see it this was a case of "failure in depth".

Just my 2 cents about various objections I've seen raised in this task:

• If anything, we should've started moving to gRPC for internal service to service communications a long time ago; we haven't done it mostly because we didn't have immediate needs.
• Our mesh is not built "around HTTP" more than it's built around gRPC. It can work pefectly well with grpc. In fact, quite a few internal functions of our service mesh are built using grpc (for the same reason it would make sense using grpc here)
• Given gRPC uses HTTP/2 for transport, and it works perfectly fine with our mesh, our ingress, and our routing logic.
• I'm not sure I understand the comment about needing an http replica, but frankly, what stops you from running your "service" (which is just a lambda) with a sidecar of the linked artifact cache to provide the http interface?

In T414338#11652561, @matmarex wrote:

@Joe @CDanis I heard you're the people to talk to about the desired data and format of these query parameters.

Currently, the proposed patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1239464 includes the following data:

• Site which is requesting the image, e.g. 'www.mediawiki.org'
• Generator (the software component involved), e.g. 'parser' or 'imageinfo'. Entry point is used as fallback if not specified, e.g. 'index', 'api', 'rest'
• Format of the requested image, 'original', 'thumbnail' or 'thumbnail_unscaled'

The format is UTM parameters (respectively utm_source, utm_campaign and utm_content, in this order), on the assumption that they'll be stripped by search engines etc.

Example: https://upload.wikimedia.org/wikipedia/commons/a/a9/Example.jpg?utm_source=mediawiki.localhost&utm_campaign=parser&utm_content=thumbnail

Your thoughts on that would be appreciated. I also have two questions:

• Do we need to sign these parameters so that they can't be spoofed, or do we start by assuming everyone will play nice? If yes, what format would be convenient? Can we just stick the data in a JWT instead of having separate parameters?
• If we're considering switching to a JWT, would it be more convenient to start with a single JSON parameter instead of separate parameters? (I mean something like https://upload.wikimedia.org/wikipedia/commons/a/a9/Example.jpg?utm_source={"site":"mediawiki.localhost","generator":"parser","format":"thumbnail"})

For now, let's remove the individual headers at the edge; we will have time to come up with a prefix naming (if any) that we want to use going forward, and to convert the headers to those values.

I don't want us to make anything under x-wmf be stripped automatically. It limits ourselves in the future.

Added an exception for translatewiki so they can keep syncing from gerrit at the previous rate, given that never created an issue for us.

Ah seen the script (thanks @Nikerabbit); first thing to do is make the user-agent compliant with the wikimedia User-Agent policy, so it should contain an email address or an url.

Do you happen to know what is the user-agent used by the script? Or point me to the source code for it.

In T364245#11635060, @tstarling wrote:

It's not using HTTPS is it?

In T402959#11542490, @Joe wrote:

In T402959#11478074, @BTracy-WMF wrote:

Hi @CDanis , we recently finalized a decision brief on next steps for this issue. We aligned upon revoking the exception and working with WDQS users to ensure their federated queries are compliant with the UA policy. To ensure we have enough lead time to communicate the upcoming change, we are requesting that the policy exception remain in place until February 18th, 2026.

Please let me know if this poses any issues on the SRE side.

Sorry, I missed this message as I was on PTO. The original agreed date was start of the year; however - February 18 is ok from our point of view, as long as there aren't further delays.

In T414805#11620283, @Ladsgroup wrote:

FWIW: Out of 9K‌ results in https://global-search.toolforge.org/?q=%5C%2Fthumb%5C%2F&regex=1&namespaces=8&title= the vast majority of them are CN banners (search for MediaWiki:Centralnotice-template) and from the remaining ones, a significant portion of them use standard sizes already.

In T414805#11612457, @Krinkle wrote:

In T414805#11612140, @gerritbot wrote:

[mediawiki/extensions/WikiEditor@master] Set background-size for toolbar buttons

https://gerrit.wikimedia.org/r/1233759

@MatthewVernon Based on discussions with SRE for T414338: [Hypothesis] WE5.4.12: Identify the provenance of image requests (FY25-26), I believe tiny icons are not our focus. Is that right? The long tail in CSS for gadgets, tools, apps, user scripts, extensions, etc to update and chase down to avoid broken UI makes this a daunting effort that I'm not sure is worth taking on righ tnow.

Could we amend the initial restriction proposed here to say that widths under 60px remain permitted? We can continue to treat these as non-standard and apply low limits to their cache misses, but we would continue to treat them as syntactically valid URLs that can be served from the cache, and at a low rate for regeneration.

Hi @Salujapushpit - I'm the project maintainer, I have seen your MR, I will review it.

I had resolved this bug quite some time ago.

Given how large the current set of actions is, this would result in a super clogged/unusable UI.

I think we have a better chance of actually integrating druid in some form within HP, rather than allowing to do this.

This has been indeed implemented already by @CDanis

Hi @Scott_French, I assume this task is resolved?

Not sure I get what you're referring to. The buttons are not javascript actions but simple submit buttons for HTML forms. They shouldn't be clickable more than once. Without reproductions steps I don't know what to do with this bug.