The SULWatchers will semi-frequently stop monitoring recent changes because of a 429 error from EventStreams. Manual intervention is then required to restart them. Sometimes, this will immediately fail with another 429, requiring a delay (5 minutes has worked for me) before restarting the RC reader again.
I don't know if the root cause is in SULWatcher specifically, or if it is a low IP-based ratelimit being applied to Toolforge.
```lines=25
2023-02-09T03:53:55Z SULWatcher MainThread INFO: starting SULWatcher
2023-02-09T03:53:55Z SULWatcher MainThread INFO: starting SULWatcher2
2023-02-09T03:53:55Z SULWatcher MainThread INFO: starting SULWatcher3
2023-02-09T03:53:55Z SULWatcher MainThread INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 48, in __init__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T03:53:59Z ib3.auth SULWatcher2 INFO: Connected to server erbium.libera.chat
2023-02-09T03:53:59Z ib3.mixins SULWatcher2 INFO: Joining #wikimedia-unifications
2023-02-09T03:54:01Z ib3.nick SULWatcher WARNING: Requested nick "SULWatcher" in use
2023-02-09T03:54:01Z ib3.nick SULWatcher3 WARNING: Requested nick "SULWatcher3" in use
2023-02-09T03:54:03Z ib3.auth SULWatcher3 INFO: Connected to server silver.libera.chat
2023-02-09T03:54:03Z ib3.mixins SULWatcher3 INFO: Joining #wikimedia-unifications
2023-02-09T03:54:03Z ib3.auth SULWatcher INFO: Connected to server silver.libera.chat
2023-02-09T03:54:03Z ib3.mixins SULWatcher INFO: Joining #wikimedia-unifications
2023-02-09T03:55:16Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T03:55:24Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T03:55:24Z SULWatcher SULWatcher INFO: AntiComposite is opped - restarting...
2023-02-09T03:55:24Z SULWatcher SULWatcher INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 48, in __init__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T04:00:51Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T04:00:51Z SULWatcher SULWatcher INFO: AntiComposite is opped - restarting...
2023-02-09T04:00:51Z SULWatcher SULWatcher INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 92, in __next__
next_chunk = next(self.resp_iterator)
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 100, in __next__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T06:25:53Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T06:25:53Z SULWatcher SULWatcher INFO: Bsadowski1 is opped - restarting...
2023-02-09T06:25:53Z SULWatcher SULWatcher INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 92, in __next__
next_chunk = next(self.resp_iterator)
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 100, in __next__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T13:48:51Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T13:49:07Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T13:49:07Z SULWatcher SULWatcher INFO: mahler is opped - restarting...
2023-02-09T13:49:07Z SULWatcher SULWatcher INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 92, in __next__
next_chunk = next(self.resp_iterator)
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 100, in __next__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T18:54:33Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T18:54:36Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T18:54:36Z SULWatcher SULWatcher INFO: AntiComposite is opped - restarting...
2023-02-09T18:54:36Z SULWatcher SULWatcher INFO: starting EventStream
Exception in thread EventsteamsThread:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 879, in run
self.listener.start()
File "/data/project/stewardbots/stewardbots/SULWatcher/SULWatcher.py", line 795, in start
for event in EventStream(url): # Listen to EventStream
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 48, in __init__
self._connect()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
File "/data/project/stewardbots/venv-k8s-py39/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange
2023-02-09T19:01:56Z SULWatcher SULWatcher INFO: do_command(self, e, 'restart rc', '#wikimedia-unifications')
2023-02-09T19:01:56Z SULWatcher SULWatcher INFO: AntiComposite is opped - restarting...
2023-02-09T19:01:56Z SULWatcher SULWatcher INFO: starting EventStream
```