After writing the ticket, I had the idea to test if it is indeed the User-Agent header that causes the 429 and it appears to make a difference.
I changed it from
CommonsFinder/1 (https://github.com/nylki/CommonsFinder) iOS 26.1.0
to
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0

for testing, and that indeed made it possible to load full images again in the app.

What puzzles me, is that when using curl command using the original User-Agent inside a terminal session, it also works:

curl -v \
        -X GET \
        -H "User-Agent: CommonsFinder/1 (https://github.com/nylki/CommonsFinder) iOS 26.1.0" \
        -H "Accept-Language: de-DE,de;q=0.9" \
        -H "Accept-Encoding: gzip, deflate, br" \
        -H "Accept: */*" \
        "https://upload.wikimedia.org/wikipedia/commons/1/18/Berlin_Mitte_June_2023_01.jpg" > Berlin_Mitte_June_2023_01.jpg

My hunch would be, that it is some heuristic against bots based on multiple factors.

Can you please report the full error message you get in the response body? Thanks

In T413570#11486635, @Joe wrote:

Can you please report the full error message you get in the response body? Thanks

Hi @Joe!
The response body is empty when this happens. So unfortunately no extra error messages to report.

I only get the header and status code.
For good measure here's the raw print of the HTTPURLResponse object (Swift/ObjC) that contains header and status code info, but it's pretty much the same as the nicely formatted log posted before (produced by the network logging library I use):

<NSHTTPURLResponse: 0x600000379e00> { URL: https://upload.wikimedia.org/wikipedia/commons/1/18/Berlin_Mitte_June_2023_01.jpg } { Status Code: 429, Headers {
    "Access-Control-Allow-Origin" =     (
        "*"
    );
    "Content-Length" =     (
        2111
    );
    "Content-Type" =     (
        "text/html; charset=utf-8"
    );
    Date =     (
        "Mon, 29 Dec 2025 16:53:23 GMT"
    );
    Server =     (
        Varnish
    );
    "Strict-Transport-Security" =     (
        "max-age=106384710; includeSubDomains; preload"
    );
    "access-control-expose-headers" =     (
        "Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache"
    );
    nel =     (
        "{ \"report_to\": \"wm_nel\", \"max_age\": 604800, \"failure_fraction\": 0.05, \"success_fraction\": 0.0}"
    );
    "report-to" =     (
        "{ \"group\": \"wm_nel\", \"max_age\": 604800, \"endpoints\": [{ \"url\": \"https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0\" }] }"
    );
    "retry-after" =     (
        1000
    );
    "server-timing" =     (
        "cache;desc=\"int-front\", host;desc=\"cp3075\""
    );
    "timing-allow-origin" =     (
        "*"
    );
    "x-analytics" =     (
        ""
    );
    "x-cache" =     (
        "cp3075 int"
    );
    "x-cache-status" =     (
        "int-front"
    );
    "x-client-ip" =     (
        redacted
    );
    "x-request-id" =     (
        "b09873b7-4248-4b0c-8203-d24b5ad3e5c9"
    );
} }

The response body is empty when this happens

That seems unlikely to me. Maybe your framework is one of those that doesn't handle response bodies when encountering errors. I remember that even 15 years ago this was already an issue with iOS apps and required custom code to able to access the data of the error response.

In T413570#11491805, @TheDJ wrote:

The response body is empty when this happens

That seems unlikely to me. Maybe your framework is one of those that doesn't handle response bodies when encountering errors. I remember that even 15 years ago this was already an issue with iOS apps and required custom code to able to access the data of the error response.

Hi @TheDJ
I'll see if I can create a minimal example project to easier reproduce and debug the issue.

For @Traffic, there have been quite a few 429 reports over the last two weeks. Reports about WhatsApp and Discord url previews not working, but also other reports on Discord and Commons VP etc. from App builders like CommonsFinder and Commons Gallery.

I originally figured this was simply due to the recent changes, but I've seen so many now that It might be worth checking if the last set of rate changes make sense and aren't accidentally applying too widely.

Apparently people are now bypassing the api limits by switching over to /thumb.php requests, so keep on eye on that as well.

@TheDJ the 429s are probably due to the massive scraping activity SREs have had to deal with over the festivities.

@Nylki I can ensure that when you get a 429 response, the response body isn't empty. In fact, it contains information that allows Wikimedia SREs to determine why you're being blocked.

hi @Joe and @TheDJ,
Thanks for sticking with me! I was indeed wrong about the empty body.
I created a minimal iOS sample app without any 3rd-party libraries to reproduce the issue, this time logging a body.
If you like, you can clone and compile yourself: https://github.com/nylki/CommonsImage429ReproApp

Here is the body (only the lower part of the html included here):

<p>Too many requests - please contact noc@wikimedia.org to discuss a less disruptive approach (3bfa1aa)</p>
</div>
</div>
<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class="text-muted"><code>Request served via cp3075 cp3075, Varnish XID 551224431<br>Upstream caches: cp3075 int<br>Error: 429, Too many requests - please contact noc@wikimedia.org to discuss a less disruptive approach (3bfa1aa) at Mon, 05 Jan 2026 15:15:19 GMT<br><details><summary>Sensitive client information</summary>IP address: redacted</details></code></p>

It is only doing a single image request, once, which should not result in a 429.
What I observed was, that the very first request was succesfull, returning image data. But on a subsequent start of the app, requests always results in a 429.

Also of interest, as observed initially:

• thumb-urls do load fine each time
• using a more common User-Agent (eg. firefox) for debugging does not yield in 429

Hi there. Chiming in as another external tool developer who also noticed the 429 issue. I develop https://commons.gallery/, a website funded by Wikimedia CH that allows Wikimedia Commons users to easily create portfolios in a Flickr-like interface. Here is an example of a typical album (this one was 13 photos).

When a user opens an album, they see smaller thumbnailed (hotlinked) versions of images in a grid. When a user clicks on a thumbnail, a lightbox opens shows a larger version of the image (either another thumbnail or full-resolution, depending on the browser width). This is similar to the on-wiki experience with Wikipedia/MediaViewer.

Since starting development, I've been trying to minimize and rate limit the number of API calls since there are of course a lot of images being displayed each time a user views an album. When a user adds a photo to their album, I cache the image data serverside (dimensions, author and license info, thumbnail URLs, etc.) so I don't have to repeat calls for that data. Only common thumbnail sizes are displayed, so we don't request weird thumbnail sizes that have never been generated before. On the clientside, I also implemented a "progressive loader" that limits the number of concurrent thumbnails that can be loaded at once.

Now, previously, the thumbnails and full-size images we were displaying were upload.wikimedia.org links. As mentioned above, I found that thumb.php serves as a workaround that I recognize is not ideal, but currently works fine without hitting any limits in my current implementation.

Anyway: around the time this ticket was opened, I also started to notice 429 errors clientside when viewing full resolution images (as upload.wikimedia.org links) in an album. Interestingly, the smaller thumbnails with a specific pixel size typically loaded fine. It's the full-size image URLs that would fail with frequency.

For example, 330px thumbnails would typically load fine without issue (despite there being dozens of these on an album page):
https://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Voidar_Koldbrann.jpg/330px-Voidar_Koldbrann.jpg

But the full size (seen when opening an image in the lightbox on desktop) would be frequently (almost always) 429ed depending on the client/user:
https://upload.wikimedia.org/wikipedia/commons/c/c5/Voidar_Koldbrann.jpg

What's weird here, from my perspective: if I were loading 50 hotlinked thumbnails at once, a 429 on some of those thumbnails wouldn't surprise me. But those tend to load fine. It's the single one-off full resolution images that fail with 429.

To emphasize, the 429s in this case are all happening clientside (so the user agent being sent is the user's own browser user agent).

Example of request headers for a failed image load:

GET /wikipedia/commons/8/8d/Emperor_at_Midgardsblot_2024.jpg HTTP/2
Host: upload.wikimedia.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0
Accept: image/avif,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Connection: keep-alive
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: cross-site
DNT: 1
Sec-GPC: 1
Priority: u=5, i
Pragma: no-cache
Cache-Control: no-cache
TE: trailers

And the response headers:

HTTP/2 429 
date: Mon, 05 Jan 2026 16:12:22 GMT
server: Varnish
x-cache: cp2034 int
x-cache-status: int-front
server-timing: cache;desc="int-front", host;desc="cp2034"
strict-transport-security: max-age=106384710; includeSubDomains; preload
report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
x-client-ip: <redacted>
access-control-allow-origin: *
access-control-expose-headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache
timing-allow-origin: *
retry-after: 1000
content-type: text/html; charset=utf-8
content-length: 2135
x-request-id: 70e5524e-1311-4104-b0cf-505382165cdf
x-analytics: 
X-Firefox-Spdy: h2

Return body:

...
<h1>Error</h1>

<p>Too many requests - please contact noc@wikimedia.org to discuss a less disruptive approach (3bfa1aa)</p>
</div>
</div>
<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class="text-muted"><code>Request served via cp2034 cp2034, Varnish XID 810352899<br>Upstream caches: cp2034 int<br>Error: 429, Too many requests - please contact noc@wikimedia.org to discuss a less disruptive approach (3bfa1aa) at Mon, 05 Jan 2026 16:46:32 GMT<br><details><summary>Sensitive client information</summary>IP address: redacted</details></code></p>
</div>
</html>

Also noting: so far it seems like I run into 429s on Firefox but not Chrome.

Finally, thanks for everything you all do to keep things running smoothly. I know media is a particular pain point with AI scrapers hammering them.

Did a bit more experimenting:

If I go to https://commons.gallery in my local Firefox and run this in the console:

fetch("https://upload.wikimedia.org/wikipedia/commons/8/8d/Emperor_at_Midgardsblot_2024.jpg")
  .then(r => r.text())
  .then(console.log);

I get a 429 back.

If I do that on any other non-Wikimedia website in the same browser, it works fine.
But if I do that on my local Chrome on commons.gallery, it works fine.

Just speculating of course, but maybe a mix of the referrer (commons.gallery) + my user agent (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0) setting off alarm bells?

Ah! I resolved the 429 issue on my end. My webapp uses Helmet.js, which defaults to hiding the Referer header when loading external files. I started allowing the Referer header to be sent when loading upload.wikimedia.org images and images are loading fine again (and I'm now back to using upload.wikimedia.org instead of thumb.php). I'm guessing with whatever recent changes were done to keep AI scrapers at bay, the Referer header became a more important heuristic.

In T413570#11494643, @SuperHamster wrote:

Ah! I resolved the 429 issue on my end. My webapp uses Helmet.js, which defaults to hiding the Referrer header when loading external files. I started allowing the Referrer header to be sent when loading upload.wikimedia.org images and images are loading fine again (and I'm now back to using upload.wikimedia.org instead of thumb.php). I'm guessing with whatever recent changes were done to keep AI scrapers at bay, the Referrer header became a more important heuristic.

That also explains why an iOS app like CommonsFinder would run into this.

In T413570#11494643, @SuperHamster wrote:

Ah! I resolved the 429 issue on my end. My webapp uses Helmet.js, which defaults to hiding the Referrer header when loading external files. I started allowing the Referrer header to be sent when loading upload.wikimedia.org images and images are loading fine again (and I'm now back to using upload.wikimedia.org instead of thumb.php). I'm guessing with whatever recent changes were done to keep AI scrapers at bay, the Referrer header became a more important heuristic.

Thanks for experimenting, good observation! Just tested, setting a referer does indeed resolve the issue. :)

Now, my 2 biggest followup questions would be:

1. Is the required referrer intended or is there still a side-effect/bug. Also, related: what other changes might be necessary to communicate to the backend that a trustworthy client is making the request.

2. what would be the preferred referer value for mobile apps, when requesting images (or any other request for that matter)? To experiment just used the description page url of the image (eg. "Referer": "https://commons.wikimedia.org/wiki/File:Berlin_Mitte_June_2023_01.jpg"), which worked. Or should it rather be some custom app url scheme (eg.: CommonsFinder://File:Berlin_Mitte_June_2023_01.jpg).

@SuperHamster Btw. I noticed that since ~ yesterday it appears to now work on my setup again without setting a referrer.

It would be interesting to hear from involved teams if there was a new relevant changeset pushed; and also to get some additional info on how best to proceed (see above) :)

EDIT: Hm, not anymore. Not sure if I looked at the wrong environment or if it was indeed a backend change.

I ran into this today, when right clicking to download an original with "Download links file as". Seems that doesn't send a referrer either. Who knew.

It does work when you click the original link and it opens it or downloads it in the browser, but that's less convenient if you already know you want to download instead of (potentially) viewing the file.

I corresponded with the e-mail address listed on the error page, and I received a response from Giuseppe Lavagetto that this was a bug. It was fixed immediately, on 12 January 2026. If this issue can no longer be reproduced, it may have been fixed at that time. I would tag Giuseppe but I do not know their username here. Possibly @Joe

Hi @Jonesey95 !
Do you have a link to the commit or ticket (if it's public)? :)

I am unfortunately still experiencing the described issue on original image files when a referer header is missing (as of 2025-01-19 (yesterday)).

It is not consistently visible though, sometimes all requested images fail to load with 429, sometimes it just works.
Contrary to a bot that usually runs on a single machine, in this case it is an app used on many users devices, the IP addresses of requests for the same User-Agent would be different in requests and vary from day to day. Could this be a factor and somehow explain why the behaviour is inconsistent from day to day?
(eg: on a some days the IPs for the User-Agent stay the same due to less test users, but on others there might be more activity).

However, as we also found out before 2026-01-12, when a referer is present, the issue does not happen or atleast does not surface to the same extent.
Does the fix only work in conjuction with a referer-header present in the http request?

thanks!

I was not sent a link to the commit. I had written an e-mail to noc@wikimedia.org, the address shown in the error message. I had no expectation that I would get anything useful in return, but my problem was acknowledged and fixed right away.

Btw. I released on new version of CommonsFinder that sends referer-headers with all network requests: https://github.com/nylki/CommonsFinder/commit/3c1727a7768b9dc3bf76d4982219f550b649832a

Will observe if the behaviour improves. Would still love to get some more insight into the topic and what the stance is on referers for desktop/mobile apps in general :) If I am not mistaken the wikipedia app does not send referer-headers.

Maybe T414048 and/or T418346 are related somehow? There seems to be an overarching theme of browser-specific 429 errors in the past few months?

In T413570#11650315, @AlexisJazz wrote:

Maybe T414048 and/or T418346 are related somehow? There seems to be an overarching theme of browser-specific 429 errors in the past few months?

To my knowledge there are now several rate-limits enforced now, that differentiate between client type (those with and without User-Agent, authenticated clients etc.), and when the rate-limit is hit 429 errors are returned.
There is one particular case, where the backend puts clients or bots in one bucket per user-agent, irregardless of the IP-Adress (or indivdual session), as far as I understand. So 10 users on 10 different devices at the same time quickly exhaust this particular rate-limit. Perhaps this is what is observed in some of those cases. I think this is the relevant changeset: https://github.com/wikimedia/operations-puppet/blob/346c3659ab04b4151566cbbe348767575d6896d0/modules/varnish/templates/text-frontend.inc.vcl.erb#L572-L579

I am definetely seeing it for CommonsFinder (the iOS mobile app), and have already sent a mail to bot-traffic@wikimedia.com, because the app appears to be classified as a bot and getting this per-user-agent rate limit, which is not optimal for an app that is used on multiple devices.

To my knowledge there are now several rate-limits enforced now, that differentiate between client type (those with and without User-Agent, authenticated clients etc.), and when the rate-limit is hit 429 errors are returned.

Here's a link to the new draft page for Wikimedia API rate limiting and these groups/types, for reference and visibility :
https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits

@SuperHamster thanks!

I am sorry to say that this issue continues to be unresolved for our iOS / Android app. There was an improvement after the initial issue was fixed but still nearly every user has 429-issues with request volumes far below (<10 / min) the rate limits (should be 200 / min) that are talked about here: https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits for clients with compliant user agent. I've send an e-mail to bot-traffic@wikimedia.com as well to see whether I was on any blacklist.

In T413570#11892868, @Bbbub wrote:

I am sorry to say that this issue continues to be unresolved for our iOS / Android app. There was an improvement after the initial issue was fixed but still nearly every user has 429-issues with request volumes far below (<10 / min) the rate limits (should be 200 / min) that are talked about here: https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits for clients with compliant user agent. I've send an e-mail to bot-traffic@wikimedia.com as well to see whether I was on any blacklist.

Indeed, still an issue, atleast for traffic on upload.wikimedia.org and with thumb images.
As far as I understand from experimentation and correspondence, it appears to be an issue with the edge layer limits in combination with identical User-Agents, irregardless of the IP-addresses (as mentioned earlier).

Thank you confirming! Did you get any actionable insights from correspondence or experimentation, like maybe adding a user-hash to the User-Agent? Or is this a problem where we need to wait for a solution?

In T413570#11893382, @Bbbub wrote:

Thank you confirming! Did you get any actionable insights from correspondence or experimentation, like maybe adding a user-hash to the User-Agent? Or is this a problem where we need to wait for a solution?

For the fullscreen/original images, setting a referer-header appears to fix the issue.
But for thumb-images, I haven't got any actionable insight. I got an answer from bot-traffic from @daniel, that the issue would be forwarded to the relevant people, but I haven't heard back in 2 months.

like maybe adding a user-hash to the User-Agent?

I considered the same. Technically it would likely fix the issue, but it I think it would violate the user-agent guidelines/rules and rather obfuscate the underlying issue. I decided to wait for a proper solution. Well, unless the backend-team says that'd would be ok.

I am currently also waiting for the OAuth2 tickets that enable support for mobile apps, and will replace the
action=clientlogin authentication as soon as possible, hoping that the situation improves atleast for authenticated users, as I understand only OAuth2 authenticated network requests get higher rate limits.

But besides from using OAuth2, it would still be good to have this issue with the per-user-agent rate limiting resolved in some way. In my opinion it clearly is a bug or an oversight of not taking mobile client architecture into account.

In T413570#11892868, @Bbbub wrote:

I am sorry to say that this issue continues to be unresolved for our iOS / Android app. There was an improvement after the initial issue was fixed but still nearly every user has 429-issues with request volumes far below (<10 / min) the rate limits (should be 200 / min) that are talked about here: https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits for clients with compliant user agent. I've send an e-mail to bot-traffic@wikimedia.com as well to see whether I was on any blacklist.

For API limits, we are going to deploy a fix for the mobile apps tomorrow. It should improve the situation. The underlying issue is that the mobile apps aren't "bots with compliant user agents". The app is really a specialized browser, and gets treated as such. At least, that#s the idea. But then, it's not quite browser-like enough sometimes, so it falls through the cracks. That's what the fix should improve.

This will however not do anything for limits on media requests.

Hi @daniel, thanks for the update :)

Thank you for the update, I'll be happy to report back if I observe any changes after tomorrow. What do you mean by

This will however not do anything for limits on media requests.

? If I am not mistaken this entire issue is about image / media requests. For us, the problem we encounter is definitely focused on the media requests from end-user clients ie apps. Are the request limits for media documented somewhere else? If not what are they if they're not counted like normal API requests?

And I am also not quite clear what you mean by

the app is really a specialized browser

? As in many apps being browser wrappers or just that you lean on some browser primitives like cookies, headers to make rate-limiting decisions?

In T413570#11895915, @Bbbub wrote:

Thank you for the update, I'll be happy to report back if I observe any changes after tomorrow. What do you mean by

This will however not do anything for limits on media requests.

? If I am not mistaken this entire issue is about image / media requests. For us, the problem we encounter is definitely focused on the media requests from end-user clients ie apps. Are the request limits for media documented somewhere else? If not what are they if they're not counted like normal API requests?

Media requests (and also requests fetching full HTML pages and other resources like CSS) are not counted as API requests. They are governed by different rules, namely the robot policy. The exact limits tend to shift quickly as we try to defend against aggressive scraping, which has been a severe problem especially for media files. I am however not an expert on this.

I supplied information about the API limits because they were discussed above.

And I am also not quite clear what you mean by

the app is really a specialized browser

? As in many apps being browser wrappers or just that you lean on some browser primitives like cookies, headers to make rate-limiting decisions?

Yes, that's what I meant - though I was primarily talking about the Wikipedia App (I suppose I misinterpreted what you meant when you mentioned "our iOS / Android app"). That's what we are targeting with the fix we are going to deploy today.

In how far CommonsFinder is (or should be) classified as a browser depends on what it does and how it does it. Right now, it is treated like a bot. This means that for unauthenticated requests, is has a single rate limit key that is shared by all clients. But most of the requests from the app are indeed authenticated... At least for API requests that is the case. Not sure how this is handled for media.

With all this said: I don't see blocked requests with "CommonsFinder" in the User-Agent string. Not for the API, and as far as I can tell also not for media.... are you sure that user-agent header is being used for media requests as well?

Thank you for the quick response! I am also not connected to CommonsFinder, our app just happens to have encountered the same problem that's why I am following this issue. Our User-Agent starts with 'by-the-ways' and it's not a browser based app. As for what it does, it shows users Points of Interests on a route. So on initial load people might see 3-8 images of Points of Interest and then more as they scroll along (but now this will almost always cause load failures if the images are hosted on Wikimedia). Our users aren't authenticated wiki-users but we should always send a user-agent. I don't think that it fits the description of robot in the robot policy but maybe that's sort of the fallback category?
If I understand correctly you can't communicate any media request limits, and the limit changes that we observed starting around the time of this issue are not a temporary aberration but here to stay, right?

Are there any other options available to us?

In T413570#11896996, @daniel wrote:

With all this said: I don't see blocked requests with "CommonsFinder" in the User-Agent string. Not for the API, and as far as I can tell also not for media.... are you sure that user-agent header is being used for media requests as well?

Yes, the User-Agent including the string "CommonsFinder" is also used for image requests that are blocked. I sent you a req/res-pair back on 2026-03-12. I attached it here now.

Right now, it is treated like a bot.

Thank you for confirming. Is there a form or another formal process to apply for non-bot status? The bot-category does not fit very well for an mobile app whose traffic is user-driven (like a browser, as you said), and not centrally orchestrated.

PS: renamed the ticket a bit, as the discussion is not really about the original/full image anymore and neither about the referrer, but in general image requests in non-WMF native apps. Feel free to revert.

In T413570#11897097, @Nylki wrote:

Right now, it is treated like a bot.

Thank you for confirming. Is there a form or another formal process to apply for non-bot status? The bot-category does not fit very well for an mobile app whose traffic is user-driven (like a browser, as you said), and not centrally orchestrated.

There is currently no good solution, but it's being worked on. I believe the person looking into the issue is @ssingh.

Thank you! Please let us know if this is also tracked or documented elsewhere so that we don't miss any developments