Page MenuHomePhabricator

Increase Lift Wing rate limit for ImpactVisualizer OAuth2 client
Closed, ResolvedPublic

Description

Wiki Education is working on an application that needs to get articlequality data for large sets of articles: https://github.com/WikiEducationFoundation/impact-visualizer

The default rate limit of Lift Wing is currently a bottleneck for the data import process of our prototype. I'd like to get access to the Wikimedia Enterprise tier for it.

Client ID: e7bca63fbc20c1bc77a5d1d347ba6d56

(CC'ing @Wittylama: this is Wiki Education's first foray into using a Wikimedia Enterprise feature. Let me know if there's any formal hoop-jumping to be done!)

Event Timeline

Thanks a lot for following up!

The wme tier is very big and created ad-hoc for the WME use case, we usually suggest to use the internal one (the name is not great) that allows 100k reqs/hour. If you need more than that, could you please add more info about the expected traffic volume and kind? For example, how many requests do you expect (bursts or sustained traffic?) and what kind of requests your tool will do (the large set of articlequality scores etc..).

internal might be fine, but since it's only 2x the default limit which we were already hitting, I thought wme would be safer.

The use case is that we pull data for a collection of articles (related to a particular topic), and then get articlequality data for each of those articles at each revision in their history (to show how the collection of articles has evolved over time). For our development and testing, we've been using a collection of about 16k articles... big enough to intentionally surface bottlenecks, but smaller than some of the collections we expect to build once this system is in production. New collections will be occasional, but when we do build a new collection and need articlequality data, that would entail hundreds of thousands of requests over the course of several hours (based on the current amount of concurrency we're using).

Change 954593 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/mediawiki-config@master] Add new OAuth Rate Limiter tier for Wiki Education

https://gerrit.wikimedia.org/r/954593

@Ragesoss thanks for the explanation! I filed a patch to create a new tier called wikieducation set for 150k request/hour (max 40 rps), that should be good for your use case. I'll let you know when deployed, so we can give it a go (regenerating the token with the new rate limit).

Change 954593 merged by jenkins-bot:

[operations/mediawiki-config@master] Add new OAuth Rate Limiter tier for Wiki Education

https://gerrit.wikimedia.org/r/954593

Mentioned in SAL (#wikimedia-operations) [2023-09-05T12:35:29Z] <elukey@deploy1002> Started scap: Backport for [[gerrit:954593|Add new OAuth Rate Limiter tier for Wiki Education (T345394)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-05T12:37:07Z] <elukey@deploy1002> elukey: Backport for [[gerrit:954593|Add new OAuth Rate Limiter tier for Wiki Education (T345394)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-09-05T12:43:18Z] <elukey@deploy1002> Finished scap: Backport for [[gerrit:954593|Add new OAuth Rate Limiter tier for Wiki Education (T345394)]] (duration: 07m 49s)

elukey@mwmaint1002:~$ mwscript extensions/OAuthRateLimiter/maintenance/setClientTierName.php --wiki metawiki --client e7bca63fbc20c1bc77a5d1d347ba6d56 --tier wikieducation
Successfully added tier wikieducation for e7bca63fbc20c1bc77a5d1d347ba6d56.

@Ragesoss you are free to regenerate your token, you should now have 150k requests/hour. Let's start with this threshold (3x the current one) and see how it goes :)