Page MenuHomePhabricator

wiki.openstreetmap.org Commons thumbs rate limit allowance
Closed, ResolvedPublic

Description

I'm part of the OpenStreetMap SRE team...

If possible, please could the wikicommons thumbs rate limit for wiki.openstreetmap.org be increased? We are frequently hitting the new throttle limits.

We (OSM Ops Team + Wiki Admins) have tried as best as possible to ensure we are only requesting standard image sizes. We run mediawiki 1.43 (LTS) which unfortunately does not yet support $wgThumbnailSteps.

Details:
wiki.openstreetmap.org - Mediawiki 1.43.8 (496f0eb) (LTS)
QuickInstantCommons 1.5.2-REL1_43 (b8e5e56)
External IP addresses: 184.104.226.103, 2001:470:1:b3b::7, 87.252.214.103, 2001:4d78:fe03:1c::7
$wgQuickInstantCommonsUserAgentInfo = "https://osm.wiki/";
Full config: https://github.com/openstreetmap/chef/blob/master/cookbooks/mediawiki/templates/default/mw-ext-QuickInstantCommons.inc.php.erb

Happy to make any suggested changes as required. If absolutely required we can upgrade to Mediawiki >=1.44.x.

Related Objects

Event Timeline

Full referrer is:

QuickInstantCommons/1.5.2-REL1_43 MediaWiki/1.43.8 OpenStreetMap%20Wiki (https://osm.wiki/)

And I see it has been throttled quite a bit lately. I wonder if caching for the extension is working as intended, as I see ~3000 requests for the same icon file in the same request, for a non-standard size in a period of 6 hours.

Aklapper renamed this task from wiki.opentreetmap.org wikicommons thumbs rate limit allowance to wiki.openstreetmap.org Commons thumbs rate limit allowance.Apr 16 2026, 1:35 PM

Is there any way for us (editors of osm wiki) to know which offending icon gets rerequested? Or even at least part of problematic file list?

With wiki reformatting and https://github.com/openstreetmap/chef/commit/729ea1372274ec067e7acb6448f5e6a42fceb0e6 it is not really clear which files are generating problematic queries

Let me ask, while all data I have access is already anonymous, it is still user's private data, just osm wiki is the referrer. Let me ask what parts (in any) I can disclose for people without an NDA.

My private email might be a good route for private data, my email is grant AT osmfoundation DOT org.
I am also happy to sign an NDA if required.

I suggest to going the route of not using the thumbnails from upload.w.o at all, if not really needed, to avoid incurring in rate-limiting that we're currently setting up to protect our media infrastructure.

If not possible at the moment I suggest to modify the configuration of QuickInstantCommons to increase descriptionCacheExpiry and apiMetadataExpiry values or introduce thumbCacheExpiry setting to increase the cache expiration time. This way you should not hit the limits so quickly...

Let us know if you have some problems in implementing this, probably someone one our side can help!

I sent an email to Grant with additional data I can share, as allowed by staff SREs.

If not possible at the moment I suggest to modify the configuration of QuickInstantCommons to increase descriptionCacheExpiry and apiMetadataExpiry values or introduce thumbCacheExpiry setting to increase the cache expiration time. This way you should not hit the limits so quickly...

Should that be apiThumbCacheExpiry, which defaults to 1 day?

If not possible at the moment I suggest to modify the configuration of QuickInstantCommons to increase descriptionCacheExpiry and apiMetadataExpiry values or introduce thumbCacheExpiry setting to increase the cache expiration time. This way you should not hit the limits so quickly...

Should that be apiThumbCacheExpiry, which defaults to 1 day?

Sorry, misspelled the option, yes it's apiThumbCacheExpiry along with apiMetadataExpiry. Setting it to avalue like 30 days should be sufficient to drastically reduce requests to our servers and not trigger this behavior

I am not seeing any 429 from this source in the last 15 days, so tentatively resolving. Please reopen if you disagree.