Page MenuHomePhabricator

Branch cut job use of proxy for Gerrit connection is causing `ProxyError`
Closed, ResolvedPublic

Description

I have no idea why this is only happening now but this error was seen after multiple attempts to test the branch cut job on releases-jenkins.

From https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut%20-%20testing%20MR%20112/4/console

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gerrit.wikimedia.org', port=443): Max retries exceeded with url: /r/a/changes/1070971/detail (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))

After investigation by myself and @hashar it seems the Puppet managed environment (including NO_PROXY) in /etc/environment.d/10-base-wmf-environment.conf does not effect the Jenkins job environments.

I confirmed that manually specifying NO_PROXY=gerrit.wikimedia.org in the job avoids the problem, so we should add that to the job definition. Using the same value that Puppet sets up makes sense.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
jenkins-rel: Configure proxy environment variables at node levelrepos/releng/jenkins-deploy!75dduvallmaster-I29e2bf95ad83dc7dbb8fd5be73d98132a561ae8emaster
Customize query in GitLab

Event Timeline

thcipriani subscribed.

Haven't seen this exact error in a bit on the job, closing optimistically.