Hi, I'm the maintainer of the WP 1.0 bot. In the past few weeks, we've been seeing a large number of exceptions originating in our calls to mwclient. They look like this:
2025-07-21 00:01:13,738:ERROR:rq.worker:[Job 1311575e-fbdd-4ad1-829a-4fba843ba6e8]: exception raised while executing (wp1.logic.project.update_project_by_name)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/rq/worker.py", line 1639, in perform_job
return_value = job.perform()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1331, in perform
self._result = self._execute()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1365, in _execute
result = self.func(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/wp1/logic/project.py", line 83, in update_project_by_name
update_project(wikidb,
File "/usr/src/app/wp1/logic/project.py", line 639, in update_project
extra_assessments = api_project.get_extra_assessments(project.p_project)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/wp1/logic/api/project.py", line 25, in get_extra_assessments
page = api.get_page(page_name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/wp1/api.py", line 62, in get_page
if not login():
^^^^^^^
File "/usr/src/app/wp1/api.py", line 46, in login
site = mwclient.Site('en.wikipedia.org',
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 129, in __init__
self.site_init()
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 149, in site_init
meta = self.get('query', meta='siteinfo|userinfo',
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 233, in get
return self.api(action, 'GET', *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 284, in api
info = self.raw_api(action, http_method, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 424, in raw_api
res = self.raw_call('api', data, retry_on_error=retry_on_error,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/mwclient/client.py", line 396, in raw_call
stream.raise_for_status()
File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: 4914820 for url: https://en.wikipedia.org/w/api.php?meta=siteinfo%7Cuserinfo%7Cuserinfo&siprop=general%7Cnamespaces&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=jsonIt looks like mwclient makes a meta=siteinfo request every time a Site object is created. I naturally realized that this might be causing a large number of unnecessary API requests and causing us to get rate limited.
However, when trying to reproduce the issue with a test, locally, I got the same exception. I hadn't been using a large number of requests from my local dev server, probably only a few total.
This leads me to wonder, is the UserAgent "mwclient/*" being rate limited particularly aggressively?
Thanks!