Page MenuHomePhabricator

cxserver: rate limiting
Closed, ResolvedPublic

Description

To provide fair service to all users of cxserver, we should implement rate limiter to not allow any single user to consume too many resources.

Some considerations for choosing a rate limiter:

  • Cxserver is currently stateless, but rate limiting requires state.
  • We can either rate limit all type of requests, or just those which consume most resources.
  • We can rate limit by the number of requests, or for example amount of characters to translate.
  • Rate limiting can be based on IP address or username for services which require authentication.

We can either implement rate limiting our selves, or we could use a existing package such as this or this if redis is available. If we want to implement it our selves, we need to create some kind of system to purge unused state to avoid indefinitely expanding storage as new users use the system. Also need to take care that the state is stored globally and not locally for each thread. Using redis avoid this problem completely, also in case we restart cxserver. The problem with the those two existing packages is that they only allow limiting by request count.

Event Timeline

Nikerabbit claimed this task.
Nikerabbit raised the priority of this task from to High.
Nikerabbit updated the task description. (Show Details)
Nikerabbit added subscribers: Arrbee, KartikMistry, santhosh and 2 others.

@GWicke, Could you please advice what is the best option here? We need this kind of gating to use MT systems that are hosted outside our cluster.

santhosh moved this task from Backlog to In Progress on the LE-Sprint-87 board.Jun 5 2015, 4:16 AM
santhosh set Security to None.Jun 9 2015, 8:05 AM
santhosh edited a custom field.
Arrbee edited projects, added LE-Sprint-88; removed LE-Sprint-87.Jun 9 2015, 9:25 AM

@santhosh, rate limiting is indeed tricky and something we need for our APIs in general. To clarify, are the requests you are interested in external (from public IPs) or internal?

Could be both, but currently the requests for MT on Special:ContentTranslation come directly from browser to cxserver.

[ Meta: Not sure how to classify it in the workboard. Bugs? Tech debt? Create a new column for Performance? Something else? @Arrbee, @Nikerabbit - suggestions are welcome. ]

Amire80 moved this task from Needs Triage to CX5 on the ContentTranslation board.Jun 23 2015, 8:15 AM
Amire80 moved this task from CX5 to CX6 on the ContentTranslation board.Jul 2 2015, 4:32 PM
Nikerabbit removed Nikerabbit as the assignee of this task.Jul 21 2015, 7:36 PM
Amire80 moved this task from CX6 to CX7 on the ContentTranslation board.Oct 2 2015, 2:45 PM
Amire80 moved this task from CX7 to CX8 on the ContentTranslation board.Jan 24 2016, 10:31 PM

Is there any update about this, now that we moved to service runner?

We did look into rate limiting options using redis, but would prefer to avoid the complexity & SPOF issues around that.

I thought we already had a tracking bug for rate limiting, but couldn't find it. So, I created T125123 to track progress.

@Amire80, @Nikerabbit: Rate limiter functionality is now part of service-runner, and is also about to be used in RESTBase. See the module documentation.

Amire80 moved this task from CX8 to Bugs on the ContentTranslation board.Apr 20 2016, 1:12 PM

As a quick update, we have been using the service-runner rate limiting functionality in production for several weeks now, and have not encountered any issues.

Danny_B removed a subscriber: Services.Jul 26 2016, 11:58 PM
GWicke lowered the priority of this task from High to Medium.Oct 12 2016, 10:28 PM
GWicke edited projects, added Services (watching); removed Services.
Arrbee moved this task from Backlog to Parsing and annotation on the CX-cxserver board.
Arrbee moved this task from Parsing and annotation to Performance on the CX-cxserver board.
santhosh closed this task as Resolved.Wed, Jan 8, 5:51 AM
santhosh claimed this task.