The current rate-limiter in Mediawiki works per IP and per user. It honors XFF headers provided by Swift. It only applies to when an image needs to be rendered, not when it already exists in Swift. It uses a DC-local cache to share the limit counts between instances.

We could implement something similar in Thumbor, but only IP-based. We should probably also differentiate "expensive" thumbnails if possible.

Sort-of related, see also T151444: Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com for hot-linked urls that result in 404s with significant rate-per-second

Yeah, I'm not sure that we rate-limit 404s actually. I'll make sure to do the right thing in Thumbor in regards to that.

I think we should be able to leverage PoolCounter for this. By locking on the IP address and leveraging PoolCounter's queue, PoolCounter would offer a very effective throttling mechanism for this. A given IP address would be allowed to generate N new thumbnails at once, and when one completes, its next request from the queue gets unlocked, etc. This allows proper throttling while still supporting legitimate traffic like opening a gallery page with hundreds of thumbnails.

Upstream PR: https://github.com/thumbor/thumbor/pull/847

Can be worked around by looking at X-Forwarded-For manually.