Page MenuHomePhabricator

Site.purgepages() does not handle rate limit
Open, Needs TriagePublic

Description

A rate limit is imposed on anyone but sysops and bureaucrats when using the purge function. The default value is max 30 pages per 60 seconds.

Right now there is nothing in Site.purgepages() which prevents it from sending too many requests nor is there anything in either it data/api.py or comms/http.py which can handle the api response other than raising it as a warning.
WARNING: API warning (purge): You've exceeded your rate limit. Please wait some time and try again

Event Timeline

As a work around:

import time
from pywikibot import config as pwb_config

batch_size = 30
rate_limit = 65  # default limit is 30 edits per 60 seconds
max_timeout = 300

# bump timeout
old_timeout = pwb_config.socket_timeout
pwb_config.socket_timeout = max_timeout

while True:
    batch = pages[:batch_size]
    pages = pages[batch_size:]
    pre_timepoint = time.time()
    result = site.purgepages(batch, **requestparams)

    if pages:
        duration = time.time()-pre_timepoint
         time.sleep(max(0, (rate_limit-duration)))
    else:
        break

# reset timeout
pwb_config.socket_timeout = old_timeout

Where pages are the pages you wish to purge on the provided site and requestparams are any params (such as forcelinkupdate) which you might want to pass on to site.purgepages().

Even this still triggers the rate limit on rare occasions but overall was quite stable.