The amount of concurrent requests from a given IP should be limited. This should work through Varnish as well (and not consider Varnish's IP from being the requesting IP).
The idea is that a given client hitting a lot of misses (whether through legitimate traffic or abuse) would get throttled automatically and not hammer Thumbor.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Use tbf vmod to throttle requests | mediawiki/vagrant | master | +207 -108 | |
| Add HAProxy in front of Thumbors | mediawiki/vagrant | master | +162 -14 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Gilles | T121388 Service-based thumbnailing re-architecture in production with Thumbor | |||
| Resolved | • Gilles | T111718 Service-based thumbnailing re-architecture on Vagrant | |||
| Resolved | • Gilles | T123461 Add client throttling to Thumbor | |||
| Resolved | • Gilles | T123660 Make Thumbor return CPU time as a header |
Change 264084 had a related patch set uploaded (by Gilles):
Add HAProxy in front of Thumbors
After discussing this with @BBlack, it seems like the best approach is to rely on the tbf vmod, which we've already invested in for production. It's disabled right now because it was causing memory leakage on VCL updates, but the idea is to keep using it in the future once those issues have been worked out.
In order to take processing cost into consideration in the rate-limiting calculations, we need a better metric than total processing time returned by Thumbor, like CPU time, which would be independent from I/O load. Which would take care of the scenario @ori described, where a user could be penalized if they happen to hit a server which is very slow because of disk issues, for example.
It seems like the xkey vmod requires 4.1 and the tbf vmod only works with 4.0 currently, not 4.1... I will try to figure out if patching for 4.1 isn't too complicated. Meanwhile, I've filed a bug: https://puszcza.gnu.org.ua/bugs/index.php?270
Comparing that other vmod's 4.0 and 4.1 branches side-by-side, the conversion doesn't seem too bad (init vs event_function):
https://github.com/varnish/libvmod-vsthrottle/blob/4.0/src/vmod_vsthrottle.c
https://github.com/varnish/libvmod-vsthrottle/blob/4.1/src/vmod_vsthrottle.c
Managed to make vmod-tbf compile properly on Varnish 4.1, made a fork for it: https://github.com/gi11es/vmod-tbf
Doesn't seem to be sufficient, sadly. When attempting to load the vmod:
vagrant@mediawiki-vagrant:/tmp/Varnish-Cache$ sudo varnishd -F -f /etc/varnish/default.vcl -a localhost:6081 Debug: Platform: Linux,3.13.0-30-generic,x86_64,-junix,-smalloc,-smalloc,-hcritbit Debug: Child (6125) Started Error: Child (6125) Pushing vcls failed: VCL "boot" Failed initialization Debug: Stopping Child Info: Child (6125) ended Could not create _.vsm.6115: File exists
In the interest of saving time, since we won't need Varnish 4 and xkey at first, I'm going to make Vagrant use Varnish 4.0 (or 3, we'll see), in order to be able to leverage the tbf vmod. Hopefully the tbf maintainer will be reasonably responsive about the 4.1 incompatibility.
Change 264084 abandoned by Gilles:
Add HAProxy in front of Thumbors
Reason:
I've managed to write a reasonably similar algorithm with the tbf vmod.
The tbf vmod author wrote the 4.1 version. Which was actually extremely close to my attempted fix, I was just missing a version parameter that indicates the vmod interface version, I believe.
Anyway, the good news is that I can have tbf and xkey at the same time now, on Varnish 4.1.
Change 265370 had a related patch set uploaded (by Gilles):
Use tbf vmod to throttle requests