Page MenuHomePhabricator

"ratelimited" API error does not throttle for rate limits
Closed, ResolvedPublic

Description

Using touch.py with my non-bot (non-admin, just autopatroller) account on Wikimedia Commons, I quickly started getting the following warnings/errors:

WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 60).jpg]] not saved.
WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 61).jpg]] not saved.
WARNING: API error ratelimited: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.
ERROR: Page [[File:Anson Technical Institute General Catalog (1972-1974) - DPLA - 6af99136b4ab3f8e0c264de148292061 (page 62).jpg]] not saved.

The attempted purges were not being saved, but Pywikibot was also not doing the normal sleeping to throttle the requests. Instead, it was just repeating the attempted purge through each page in the list without any slowing.

My guess is that this is caused by the purge rate limit being lower than the edit rate limit, and touch.py not handling that correctly?

Event Timeline

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMay 20 2020, 1:05 AM
Xqt triaged this task as Medium priority.May 20 2020, 3:49 AM
Xqt added a subscriber: Multichill.May 20 2020, 3:56 AM

Change 598269 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] Throttle requests after ratelimits exceeded

https://gerrit.wikimedia.org/r/598269

Xqt renamed this task from touch.py does not throttle for rate limits correctly to "ratelimited" API error does not throttle for rate limits.May 24 2020, 6:20 PM
Xqt claimed this task.
JJMC89 added a subscriber: JJMC89.May 24 2020, 7:29 PM

Re 598269 PS2:
The patch iterates a specific set of groups and calculates the delay based on the first one hit.

It should iterate the groups in the response and calculate the delay and wait for the smallest duration.

Using the example below response below for the move action:
user: delay=60/8=7.5
autopatrolled: delay = 60/32=1.875
therefore wait for 1.875 seconds

Example response (my Commons rate limits)
{
    "batchcomplete": "",
    "query": {
        "userinfo": {
            "id": 4724680,
            "name": "JJMC89",
            "ratelimits": {
                "move": {
                    "user": {
                        "hits": 8,
                        "seconds": 60
                    },
                    "autopatrolled": {
                        "hits": 32,
                        "seconds": 60
                    }
                },
                "edit": {
                    "user": {
                        "hits": 900,
                        "seconds": 180
                    },
                    "autopatrolled": {
                        "hits": 10500,
                        "seconds": 180
                    }
                },
                "upload": {
                    "user": {
                        "hits": 380,
                        "seconds": 4320
                    },
                    "autopatrolled": {
                        "hits": 999,
                        "seconds": 1
                    }
                },
                "linkpurge": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "badcaptcha": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "emailuser": {
                    "user": {
                        "hits": 20,
                        "seconds": 86400
                    }
                },
                "changeemail": {
                    "user": {
                        "hits": 4,
                        "seconds": 86400
                    }
                },
                "rollback": {
                    "user": {
                        "hits": 100,
                        "seconds": 60
                    }
                },
                "purge": {
                    "user": {
                        "hits": 30,
                        "seconds": 60
                    }
                },
                "renderfile": {
                    "user": {
                        "hits": 700,
                        "seconds": 30
                    }
                },
                "renderfile-nonstandard": {
                    "user": {
                        "hits": 70,
                        "seconds": 30
                    }
                },
                "cxsave": {
                    "user": {
                        "hits": 10,
                        "seconds": 30
                    }
                },
                "urlshortcode": {
                    "user": {
                        "hits": 50,
                        "seconds": 120
                    }
                },
                "thanks-notification": {
                    "user": {
                        "hits": 10,
                        "seconds": 60
                    }
                },
                "badoath": {
                    "user": {
                        "hits": 10,
                        "seconds": 60
                    }
                }
            }
        }
    }
}
Xqt added a comment.May 24 2020, 8:49 PM

Many thanks for this response sample

Change 598269 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] Throttle requests after ratelimits exceeded

https://gerrit.wikimedia.org/r/598269

JJMC89 closed this task as Resolved.May 25 2020, 5:15 PM