Pywikibot eventstreams.py 504 Server Error: Gateway Timeout
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	TheSandDoctor
	Mar 11 2020, 7:48 PM

Description

I did not encounter this previously and have made no substantial code changes, but by only script hooked into the event stream keeps crashing with 504 timeout errors. The script in question: https://github.com/TheSandDoctor/Commons-image-corruption-detector/blob/master/rcwatcher.py

Traceback (most recent call last):
  File "rcwatcher.py", line 65, in <module>
    main()
  File "rcwatcher.py", line 57, in main
    run_watcher()
  File "rcwatcher.py", line 41, in run_watcher
    for change in rc:
  File "/usr/local/lib/python3.8/site-packages/pywikibot/comms/eventstreams.py", line 291, in __iter__
    self.source = EventSource(**self.sse_kwargs)
  File "/home/thesanddoctor/sseclient/sseclient.py", line 48, in __init__
    self._connect()
  File "/home/thesanddoctor/sseclient/sseclient.py", line 63, in _connect
    self.resp.raise_for_status()
  File "/home/ccc/.local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: https://stream.wikimedia.org/v2/stream/recentchange

Related Objects

Mentioned Here: T238658: Migrate EventStreams to k8s deployment pipeline

Event Timeline

TheSandDoctor created this task.Mar 11 2020, 7:48 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMar 11 2020, 7:48 PM

TheSandDoctor added subscribers: zhuyifei1999, Dvorapa.Mar 11 2020, 7:52 PM

This is happening only a few seconds after I start the script.

Why do you assume that 504 error isn’t real? Which sseclient are you using?

I am using the latest sseclient. I assume that it isn't "real" as event streams shouldn't be 504ing constantly? That wouldn't make sense & would mean that recent changes should be broken etc? This script is connecting to it through pywikibot's. site_rc_listener().

Probably streams server tries to access wikidata :D - anyway, looks like an upstream issue

Xqt added a project: Phabricator (Upstream).Mar 12 2020, 6:25 AM

Xqt added a subscriber: Ottomata.

Restricted Application added a project: Upstream. · View Herald TranscriptMar 12 2020, 6:25 AM

Xqt removed a project: Phabricator (Upstream).Mar 12 2020, 6:26 AM

There are also other timeouts found:

Store event in buffer with length 6
HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Read timed out.
WARNING: Connection error: 502 Server Error: connect failed for url: https://stream.wikimedia.org/v2/stream/recentchange
Try to re-establish connection.

1897 pages read
1075 pages written
0 pages skipped
Execution time: 1 days, 5860 seconds
Read operation time: 48 seconds
Write operation time: 85 seconds
Script terminated by exception:

ERROR: HTTPError: 502 Server Error: connect failed for url: https://stream.wikimedia.org/v2/stream/recentchange
Traceback (most recent call last):
  File "C:\pwb\core\pwb.py", line 325, in <module>
    if not main():
  File "C:\pwb\core\pwb.py", line 320, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "C:\pwb\core\pwb.py", line 101, in run_python_file
    main_mod.__dict__)
  File ".\scripts\redirect.py", line 921, in <module>
    main()
  File ".\scripts\redirect.py", line 917, in main
    bot.run()
  File "C:\pwb\core\pywikibot\bot.py", line 1724, in run
    super(MultipleSitesBot, self).run()
  File "C:\pwb\core\pywikibot\bot.py", line 1499, in run
    for item in self.generator:
  File ".\scripts\redirect.py", line 370, in retrieve_double_redirects
    for event in stream:
  File "C:\pwb\core\pywikibot\comms\eventstreams.py", line 293, in __iter__
    self.source = EventSource(**self.sse_kwargs)
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 40, in __init__
    self._connect()
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 53, in _connect
    self.resp.raise_for_status()
  File "C:\Python37-32\lib\site-packages\requests\models.py", line 940, in raise
_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: connect failed for url: https:/
/stream.wikimedia.org/v2/stream/recentchange
CRITICAL: Exiting due to uncaught exception <class 'requests.exceptions.HTTPError'>

>>> Fixing double redirects on wikipedia:it <<<
HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Read timed out.
WARNING: Connection error: HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Max retries exceeded with url: /v2/stream/recentchange (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x080D3730>:
Failed to establish a new connection: [WinError 10061] Es konnte keine Verbindung hergestellt werden, da der Zielcomputer die Verbindung verweigerte'))
Try to re-establish connection.

27531 pages read
18287 pages written
0 pages skipped
Execution time: 11 days, 31520 seconds
Read operation time: 35 seconds
Write operation time: 53 seconds
Script terminated by exception:

ERROR: ConnectTimeout: HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Max retries exceeded with url: /v2/stream/recentchange (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x07FB15F0>, 'Connection to stream.wikimedia.org timed out. (connect timeout=6.05)'))
Traceback (most recent call last):
  File "C:\Python37-32\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "C:\Python37-32\lib\site-packages\urllib3\util\connection.py", line 80, in create_connection
    raise err
  File "C:\Python37-32\lib\site-packages\urllib3\util\connection.py", line 70, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "C:\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "C:\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "C:\Python37-32\lib\site-packages\urllib3\connection.py", line 316, in connect
    conn = self._new_conn()
  File "C:\Python37-32\lib\site-packages\urllib3\connection.py", line 165, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnec
tion object at 0x07FB15F0>, 'Connection to stream.wikimedia.org timed out. (connect timeout=6.05)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Python37-32\lib\site-packages\urllib3\util\retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Max retries exceeded with url: /v2/stream/recentchange (Caused by
ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x07FB15F0>, 'Connection to stream.wikimedia.org timed out. (connect timeout=6.05)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\pwb\core\pwb.py", line 325, in <module>
    if not main():
  File "C:\pwb\core\pwb.py", line 320, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "C:\pwb\core\pwb.py", line 101, in run_python_file
    main_mod.__dict__)
  File ".\scripts\redirect.py", line 921, in <module>
    main()
  File ".\scripts\redirect.py", line 917, in main
    bot.run()
  File "C:\pwb\core\pywikibot\bot.py", line 1728, in run
    """Reset the bot's site after run."""
  File "C:\pwb\core\pywikibot\bot.py", line 1501, in run
    try:
  File ".\scripts\redirect.py", line 370, in retrieve_double_redirects
    for event in stream:
  File "C:\pwb\core\pywikibot\comms\eventstreams.py", line 293, in __iter__
    self.source = EventSource(**self.sse_kwargs)
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 40, in __init__
    self._connect()
  File "C:\Python37-32\lib\site-packages\sseclient.py", line 48, in _connect
    self.resp = requester.get(self.url, stream=True, **self.requests_kwargs)
  File "C:\Python37-32\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python37-32\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python37-32\lib\site-packages\requests\adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='stream.wikimedia.org', port=443): Max retries exceeded with url: /v2/stream/recentchange (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x07FB15F0>, 'Connection to stream.wikimedia.org timed out. (connect timeout=6.05)'))
CRITICAL: Exiting due to uncaught exception <class 'requests.exceptions.ConnectTimeout'>

Waiting for events....('Connection broken: IncompleteRead(0 bytes read)', Incomp
leteRead(0 bytes read))
......('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes
 read))
WARNING: Empty message found.
.('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read
))
WARNING: Empty message found.
..........('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 b
ytes read))
WARNING: Empty message found.
Waiting for events..('Connection broken: IncompleteRead(0 bytes read)', Incomple
teRead(0 bytes read))
WARNING: Empty message found.
.
Found a new edit by user "<redacted>"

Same for our tests:

Xqt added a project: EventStreams.Mar 12 2020, 11:48 AM

Ah I'm sorry! This was caused by https://phabricator.wikimedia.org/T238658#5963534. All my tests indicated that while clients were being disconnected, they were reconnecting ok so we tolerated the timeouts until we figured out how to fix this morning.

I've just applied the fix in eqiad (codfw was already done), so hopefully this should be better now. I will be watching this closely today.

In T247447#5963778, @Ottomata wrote:

I've just applied the fix in eqiad (codfw was already done), so hopefully this should be better now. I will be watching this closely today.

@Ottomata: Thanks for explantion and fixing this issue.
Appveyor tests passed this morning https://ci.appveyor.com/project/Ladsgroup/pywikibot-g4xqx/builds/31418040
But Travis still failed. I'll restart the build to test there shortenly.

Looks good overall: https://travis-ci.org/github/wikimedia/pywikibot/builds/660870636
The remaining issues are releated to wikidata which gives empty results.

Xqt removed Xqt as the assignee of this task.Mar 12 2020, 2:16 PM

Pywikibot eventstreams.py 504 Server Error: Gateway TimeoutClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Pywikibot eventstreams.py 504 Server Error: Gateway Timeout
Closed, ResolvedPublic
Actions