As mentioned in discussion in T90115, we may want to make nginx to limit connection rate per client IP. Depends on external layers passing us the connecting IP.
See also http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html and http://nginx.org/en/docs/http/ngx_http_realip_module.html
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Limit concurrent connections by client IP | operations/puppet | production | +14 -1 |
To really work this needs some sort of global state per IP (or/and sharding) otherwise each IP may still use a fraction of each back end server. Other things use User::pingLimiter() (from mediawiki core, uses increments on memcached keys) or https://wikitech.wikimedia.org/wiki/PoolCounter .
The limit conn nginx module would in this case only limit the connections for each back end server separately.
I think for now it's premature to try and build a full-blown DoS resistant system. I'm not even sure we need per-IP limits, but if we do, unless we're dealing with real DoS, per-backend IP protection probably would be enough. If we get real DoS problem, we'd probably want to solve it on frontend level and not for each backend separately.
Change 319010 had a related patch set uploaded (by Smalyshev):
Limit concurrent connections by client IP
I propose to start by enabling connection limiting with a fairly high limit and lowering it after a week, once we get some feedback.
A quick analysis of the situation after 2 days of limiting connections:
Now that we have some data, it might make sense to do some more analysis to see which users we are blocking and see if that is an issue (at least no one has been screaming yet... but that's not enough of a validation). @Smalyshev, @mpopov, do you have any idea how to approach this analysis?
@Gehel Can we see if the limited connections are from same IPs or different IPs? I.e., how many IPs are affected and how many each of them got limited?