"ratelimited" API error does not throttle for rate limits
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dominicbm
	May 20 2020, 1:05 AM

Description

Using touch.py with my non-bot (non-admin, just autopatroller) account on Wikimedia Commons, I quickly started getting the following warnings/errors:

WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 60).jpg]] not saved.
WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 61).jpg]] not saved.
WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 62).jpg]] not saved.

The attempted purges were not being saved, but Pywikibot was also not doing the normal sleeping to throttle the requests. Instead, it was just repeating the attempted purge through each page in the list without any slowing.

My guess is that this is caused by the purge rate limit being lower than the edit rate limit, and touch.py not handling that correctly?

Details

	Subject	Repo	Branch	Lines +/-
	[IMPR] Throttle requests after ratelimits exceeded	pywikibot/core	master	+27 -5

Customize query in gerrit

Related Objects

Mentioned In: T270912: "ratelimited" API error does not wait to retry for purge
T268645: Pywikibot crashing on ratelimit after the ratelimit removal from bot group in 2020
rPWBCcaca33889b09: [IMPR] Throttle requests after ratelimits exceeded
Mentioned Here: T192912: Unexpected ratelimit makes Pywikibot crash

Event Timeline

Dominicbm created this task.May 20 2020, 1:05 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMay 20 2020, 1:05 AM

See also T192912

Xqt triaged this task as Medium priority.May 20 2020, 3:49 AM

Xqt added a subscriber: Multichill.May 20 2020, 3:56 AM

Change 598269 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] Throttle requests after ratelimits exceeded

https://gerrit.wikimedia.org/r/598269

gerritbot added a project: Patch-For-Review.May 24 2020, 6:19 PM

Xqt renamed this task from touch.py does not throttle for rate limits correctly to "ratelimited" API error does not throttle for rate limits.May 24 2020, 6:20 PM

Xqt claimed this task.

Re 598269 PS2:
The patch iterates a specific set of groups and calculates the delay based on the first one hit.

It should iterate the groups in the response and calculate the delay and wait for the smallest duration.

Using the example below response below for the move action:
user: delay=60/8=7.5
autopatrolled: delay = 60/32=1.875
therefore wait for 1.875 seconds

Example response (my Commons rate limits)

{
    "batchcomplete": "",
    "query": {
        "userinfo": {
            "id": 4724680,
            "name": "JJMC89",
            "ratelimits": {
                "move": {
                    "user": {
                        "hits": 8,
                        "seconds": 60
                    },
                    "autopatrolled": {
                        "hits": 32,
                        "seconds": 60
                    }
                },
                "edit": {
                    "user": {
                        "hits": 900,
                        "seconds": 180
                    },
                    "autopatrolled": {
                        "hits": 10500,
                        "seconds": 180
                    }
                },
                "upload": {
                    "user": {
                        "hits": 380,
                        "seconds": 4320
                    },
                    "autopatrolled": {
                        "hits": 999,
                        "seconds": 1
                    }
                },
                "linkpurge": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "badcaptcha": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "emailuser": {
                    "user": {
                        "hits": 20,
                        "seconds": 86400
                    }
                },
                "changeemail": {
                    "user": {
                        "hits": 4,
                        "seconds": 86400
                    }
                },
                "rollback": {
                    "user": {
                        "hits": 100,
                        "seconds": 60
                    }
                },
                "purge": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "renderfile": {
                    "user": {
                        "hits": 700,
                        "seconds": 30
                    }
                },
                "renderfile-nonstandard": {
                    "user": {
                        "hits": 70,
                        "seconds": 30
                    }
                },
                "cxsave": {
                    "user": {
                        "hits": 10,
                        "seconds": 30
                    }
                },
                "urlshortcode": {
                    "user": {
                        "hits": 50,
                        "seconds": 120
                    }
                },
                "thanks-notification": {
                    "user": {
                        "hits": 10,
                        "seconds": 60
                    }
                },
                "badoath": {
                    "user": {
                        "hits": 10,
                        "seconds": 60
                    }
                }
            }
        }
    }
}