Page MenuHomePhabricator

`scap backport` should gracefully handle a single error when polling for patch merge
Closed, ResolvedPublic

Description

Saw this today:

13:35:48 Change 835590 approved
13:35:49 Change 835591 approved
13:35:49 Waiting for changes to be merged. This may take some time if there are long running tests.
13:36:59 awaiting-backport-merges:   0% (ok: 0; fail: 0; left: 2)               
13:36:59 Unhandled error:
Traceback (most recent call last):
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1352, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/var/lib/scap/scap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1352, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/cli.py", line 524, in run
    exit_status = app.main(app.extra_arguments)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/backport.py", line 106, in main
    self._do_backport(change_numbers, change_details)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/backport.py", line 136, in _do_backport
    self._wait_for_changes_to_be_merged(change_numbers)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/backport.py", line 448, in _wait_for_changes_to_be_merged
    detail = self.gerrit.change_detail(number).get()
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/gerrit.py", line 170, in get
    res = self._session.get(uri, params=params, timeout=30)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/gerrit.py", line 124, in get
    return self.session.get(*args, **kwargs)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
13:36:59 backport failed: <ConnectionError> ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Wondering if scap should just ignore a single connection error if the next request succeeds?

Event Timeline

Change 835650 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/scap@master] Enable retry support for Gerrit HTTP requests

https://gerrit.wikimedia.org/r/835650

Change 835650 merged by jenkins-bot:

[mediawiki/tools/scap@master] Enable retry support for Gerrit HTTP requests

https://gerrit.wikimedia.org/r/835650

Included in scap 4.24.0 which was deployed today.