Page MenuHomePhabricator

Add Retry-After header when a rate limit has been triggered
Open, Needs TriagePublic

Description

When using a bot if a rate limit is triggered there is no programmatic way I could found to know how many time I should wait to retry and continue with the task. Would it be possible to add a Retry-After header in this cases with the amount of time the bot should wait before retrying the request?

Additionally, it would be nice if the response status would indicate that the operation wasn't completed successfully not using the 200 OK status code. Semantically the most appropriate one would probably be 429 Too many requests, which based on the RFC 6585 may use the Retry-After header.

Event Timeline

Agabi10 created this task.Aug 2 2017, 3:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 2 2017, 3:25 PM
Anomie added a subscriber: Anomie.Aug 2 2017, 10:22 PM

When using a bot if a rate limit is triggered there is no programmatic way I could found to know how many time I should wait to retry and continue with the task. Would it be possible to add a Retry-After header in this cases with the amount of time the bot should wait before retrying the request?

The underlying code typically used for rate limits doesn't return any "time to wait" information, so that would need to be changed first before the API could indicate it.

Additionally, it would be nice if the response status would indicate that the operation wasn't completed successfully not using the 200 OK status code.

The API basically uses HTTP as a transport, returning errors at its own layer rather than the transport layer. See T40716: API should use status codes for errors for more.

Anomie moved this task from Unsorted to Blocked on the MediaWiki-API board.Aug 2 2017, 10:22 PM
Gstupp added a subscriber: Gstupp.May 29 2018, 11:35 PM
Xqt added a subscriber: Xqt.Nov 28 2018, 1:24 PM
Restricted Application added a project: Platform Engineering. · View Herald TranscriptApr 7 2020, 3:40 PM
TheDJ added a subscriber: TheDJ.Jul 29 2020, 11:41 AM

I was helping some users who ran into rate limits and we honestly do a bad job at explaining rate limits to api users. The API documentation was even pretending like there wasn't even a rate limit (likely because originally written with a focus on read action usage, but we have also simple added and raised a lot of rate limits over the past year, so ppl actually tend to run into them now).
I have added some extra information to: https://www.mediawiki.org/wiki/API:Etiquette#Request_limit which hopefully makes it at least a bit more discoverable.

But I also noticed that we don't tell you how long to wait, which seems like a pretty basic functionality of a rate limited api... I do think we need to do better, and I am in favour of Retry-After: header, mostly because maxlag also emits Retry-After, so that makes implementation just a whole lot more sensible in my opinion.

However, in order to add a Retry-After header, like Brad said, we need that information. That means
1: Extending ThrottledError with rate information
2: Extending pingLimiter return value with rate information
3: Forward this rate information through the status object to the api handler

Izno added a subscriber: Izno.Aug 13 2020, 1:26 AM

The docs situation about rate limits has also been brought up in https://www.mediawiki.org/wiki/Topic:Vueghdcw4pexplhv
Boldly adding Documentation per last comment.

Yug added a subscriber: Yug.EditedSep 23 2020, 6:06 PM

The docs situation about rate limits has also been brought up in https://www.mediawiki.org/wiki/Topic:Vueghdcw4pexplhv
Boldly adding Documentation per last comment.

MediaWiki API should have a query returning all the ratelimits, for each action, all existing userrights, and their ratelimit. Should we open a ticket on this ?

Backstory: A LinguaLibre (WM-fr) contributor bumped into ratelimit issue in June, lost about 500+ audio recordings. The console log error was really minimalist, hinting at ratelimit. MediaWiki documentation on ratelimit being nearly existent, I failed to find my answer in June. I did a new round of research around Sept. 20th, finding sparse informations. I eventually asked various users with various userrights to ping Commons API via :

so I could little by little rebuild the full ratelimit profile of Commons. The other way was that I found it in Phabricator some ratelimit discussions and then, which hinted at in-code settings, which I eventually also found after a fastidious quest : https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/InitialiseSettings.php#9983

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:01 PM