Page MenuHomePhabricator

Implement rate limiter in Thumbor
Closed, ResolvedPublic

Event Timeline

Gilles created this task.Nov 18 2016, 4:42 PM

The current rate-limiter in Mediawiki works per IP and per user. It honors XFF headers provided by Swift. It only applies to when an image needs to be rendered, not when it already exists in Swift. It uses a DC-local cache to share the limit counts between instances.

We could implement something similar in Thumbor, but only IP-based. We should probably also differentiate "expensive" thumbnails if possible.

Gilles renamed this task from Implement rate limiter to Implement rate limiter in Thumbor.Nov 23 2016, 12:14 PM

Sort-of related, see also T151444: Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com for hot-linked urls that result in 404s with significant rate-per-second

elukey removed a subscriber: elukey.Nov 24 2016, 11:31 AM

Yeah, I'm not sure that we rate-limit 404s actually. I'll make sure to do the right thing in Thumbor in regards to that.

Gilles added a comment.EditedDec 22 2016, 8:54 PM

I think we should be able to leverage PoolCounter for this. By locking on the IP address and leveraging PoolCounter's queue, PoolCounter would offer a very effective throttling mechanism for this. A given IP address would be allowed to generate N new thumbnails at once, and when one completes, its next request from the queue gets unlocked, etc. This allows proper throttling while still supporting legitimate traffic like opening a gallery page with hundreds of thumbnails.

Gilles added a comment.Jan 2 2017, 4:27 PM

Upstream PR: https://github.com/thumbor/thumbor/pull/847

Can be worked around by looking at X-Forwarded-For manually.

Gilles closed this task as Resolved.Feb 1 2017, 7:49 PM