Error 429: too many requests for stream.wikimedia.org
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	Xqt
	May 21 2022, 4:56 PM

Description

List of steps to reproduce:
running a bot script with EventStreams fails upstream with http Error 429

What happens?:

Execution time: 0 seconds
Script terminated by exception:

ERROR: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org
/v2/stream/recentchange (HTTPError)
Traceback (most recent call last):
  File "C:\pwb\core\pwb.py", line 487, in <module>
    main()
  File "C:\pwb\core\pwb.py", line 471, in main
    if not execute():
  File "C:\pwb\core\pwb.py", line 454, in execute
    run_python_file(filename, script_args, module)
  File "C:\pwb\core\pwb.py", line 144, in run_python_file
    main_mod.__dict__)
  File ".\scripts\redirect.py", line 906, in <module>
    main()
  File ".\scripts\redirect.py", line 902, in main
    bot.run()
  File "C:\pwb\core\pywikibot\bot.py", line 1628, in run
    for item in self.generator:
  File "C:\pwb\core\pywikibot\bot.py", line 1626, in <genexpr>
    self.generator = (item for item in self.generator)
  File ".\scripts\redirect.py", line 367, in retrieve_double_redirects
    for event in stream:
  File "C:\pwb\core\pywikibot\comms\eventstreams.py", line 283, in __iter__
    self.source = EventSource(**self.sse_kwargs)
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 40, in __init__
    self._connect()
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 53, in _connect
    self.resp.raise_for_status()
  File "C:\Python37-32\lib\site-packages\requests\models.py", line 940, in raise
_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: http
s://stream.wikimedia.org/v2/stream/recentchange
CRITICAL: Exiting due to uncaught exception <class 'requests.exceptions.HTTPErro
r'>

Software version:7
Python 3.7.3
Pywikibot 7.4.0.dev0
requests 2.22.0
sseclient: 0.0.22

Related Objects

Mentioned In: T363296: toolforge: explore options to introduce egress network quotas
T329327: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher
Mentioned Here: T329327: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher

Event Timeline

Xqt created this task.May 21 2022, 4:56 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMay 21 2022, 4:56 PM

This happpens when an IP opens up too many simultaneous connections to the service. The client IP is passed on to the eventstreams service from the Varnish frontend http servers, and then a running count of connections per eventstreams application instance is kept. If too many connections are made to that application instance, a 429 is returned. The current limit is connections 2 per application instance, of which there are 8 in each DC. So you should be able to open a total of 16 connections from your IP to a single DC (you will be routed to one of the two automatically based on your location). If you get this error, and are opening fewer than 16 connections, I'd expect a reconnect to route eventually route you to an application instance where you have fewer than 2 connections open.

I suppose, the more connections you have open at once, the more likely this is to happen.

I suppose, the more connections you have open at once, the more likely this is to happen.

I only had one connection open as this happened.

Maybe unlikely, but since it is per client IP, is it at all possible you were behind a NAT that might also have other connections to stream.wikimedia.org?

Did reconnecting help? Is this reproducible now? What is the client IP you are connecting from?

I've checked my logs. There was only one connection started 16:38:25 UTC and failed immediately after starting start. Next start were mad 17:07:37 and a second connection was established 17:21:07, 17:22:35, 17:22:53 for tests. I can rule it out that other instances were running behind the same IP. This issue was only temporary and I haven't seen it again in meantime or ever before. Maybe we can close it.

See also T329327

This error still occures. I have 3 intances running eventstreams but got only one failing:

Get new event from stream
Store event in buffer with length 6
Get new event from stream
Store event in buffer with length 7
Get new event from stream
>>> buffer 983
Store event in buffer with length 8
("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
WARNING: Connection error: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange.
Try to re-establish connection.

3059 read operations
2803 write operations
Execution time: 1 days, 43376 seconds
Read operation time: 42.4 seconds
Write operation time: 46.3 seconds
Script terminated by exception:

ERROR: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange (HTTPError)
Traceback (most recent call last):
  File "D:\pwb\core\pwb.py", line 39, in <module>
    sys.exit(main())
             ^^^^^^
  File "D:\pwb\core\pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "D:\pwb\core\pywikibot\scripts\wrapper.py", line 516, in <module>
    main()
  File "D:\pwb\core\pywikibot\scripts\wrapper.py", line 500, in main
    if not execute():
           ^^^^^^^^^
  File "D:\pwb\core\pywikibot\scripts\wrapper.py", line 487, in execute
    run_python_file(filename, script_args, module)
  File "D:\pwb\core\pywikibot\scripts\wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "D:\pwb\core\scripts\redirect.py", line 899, in <module>
    main()
  File "D:\pwb\core\scripts\redirect.py", line 895, in main
    bot.run()
  File "D:\pwb\core\pywikibot\bot.py", line 1686, in run
    for item in self.generator:
  File "D:\pwb\core\pywikibot\bot.py", line 1684, in <genexpr>
    self.generator = (item for item in self.generator)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\core\scripts\redirect.py", line 366, in retrieve_double_redirects
    for event in stream:
  File "<frozen _collections_abc>", line 330, in __next__
  File "D:\pwb\core\pywikibot\tools\collections.py", line 275, in send
    return next(self._started_gen)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\core\pywikibot\comms\eventstreams.py", line 326, in generator
    self.source = EventSource(**self.sse_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\sseclient.py", line 40, in __init__
    self._connect()
  File "C:\Python311\Lib\site-packages\sseclient.py", line 53, in _connect
    self.resp.raise_for_status()
  File "C:\Python311\Lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
CRITICAL: Exiting due to uncaught exception HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange

Ottomata added projects: Event-Platform, Data Engineering and Event Platform Team.Sep 22 2023, 3:24 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptSep 22 2023, 3:24 PM

lbowmaker removed a project: Data Engineering and Event Platform Team.Nov 10 2023, 2:48 PM

lbowmaker moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Feb 16 2024, 8:57 PM

bd808 mentioned this in T329327: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher.Tue, Apr 23, 10:44 PM

aborrero mentioned this in T363296: toolforge: explore options to introduce egress network quotas.Wed, Apr 24, 8:09 AM

Error 429: too many requests for stream.wikimedia.orgOpen, Needs TriagePublicBUG REPORTActions

Description

Related Objects

Event Timeline

Error 429: too many requests for stream.wikimedia.org
Open, Needs TriagePublicBUG REPORT
Actions