|Resolved||Gilles||T121388 Service-based thumbnailing re-architecture in production with Thumbor|
|Resolved||Gilles||T151067 Implement rate limiter in Thumbor|
The current rate-limiter in Mediawiki works per IP and per user. It honors XFF headers provided by Swift. It only applies to when an image needs to be rendered, not when it already exists in Swift. It uses a DC-local cache to share the limit counts between instances.
We could implement something similar in Thumbor, but only IP-based. We should probably also differentiate "expensive" thumbnails if possible.
Sort-of related, see also T151444: Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com for hot-linked urls that result in 404s with significant rate-per-second
I think we should be able to leverage PoolCounter for this. By locking on the IP address and leveraging PoolCounter's queue, PoolCounter would offer a very effective throttling mechanism for this. A given IP address would be allowed to generate N new thumbnails at once, and when one completes, its next request from the queue gets unlocked, etc. This allows proper throttling while still supporting legitimate traffic like opening a gallery page with hundreds of thumbnails.