Page MenuHomePhabricator
Create Task
Maniphest T114650

tox-jessie job is failing randomly on integration/config with could not install deps, because of throttling
Closed, ResolvedPublic
Actions

Description

The tox-jessie job is failing randomly on integration/config with could not install deps. This is because PiPy is throttling us.

https://integration.wikimedia.org/ci/job/tox-jessie/224/console

15:08:44 py27 installdeps: -rtest-requirements.txt
15:08:44   /home/jenkins/workspace/tox-jessie$ /home/jenkins/workspace/tox-jessie/.tox/py27/bin/pip install -rtest-requirements.txt >/home/jenkins/workspace/tox-jessie/.tox/py27/log/py27-1.log
15:09:09 ERROR: invocation failed (exit code 2), logfile: /home/jenkins/workspace/tox-jessie/.tox/py27/log/py27-1.log
15:09:09 ERROR: actionid=py27
15:09:09 msg=getenv
15:09:09 cmdargs=[local('/home/jenkins/workspace/tox-jessie/.tox/py27/bin/pip'), 'install', '-rtest-requirements.txt']
15:09:09 env={'JENKINS_HOME': '/var/lib/jenkins', 'OFFLINE_NODE_WHEN_COMPLETE': '1', 'SAUCE_ONDEMAND_USERNAME': 'wikimedia-jenkins', 'HHVM_REPO_CENTRAL_PATH': '/home/jenkins/workspace/tox-jessie/central.hhbc', 'HUDSON_URL': 'https://integration.wikimedia.org/ci/', 'BUILD_URL': 'https://integration.wikimedia.org/ci/job/tox-jessie/224/', 'GIT_PREVIOUS_COMMIT': '12726579b2057bda6a1b715db605303c4c0c55cc', 'BUILD_TAG': 'jenkins-tox-jessie-224', 'SSH_CLIENT': '208.80.154.135 62908 22', 'JENKINS_URL': 'https://integration.wikimedia.org/ci/', 'LOGNAME': 'jenkins', 'USER': 'jenkins', 'WORKSPACE': '/home/jenkins/workspace/tox-jessie', 'HOME': '/home/jenkins', 'PATH': '/home/jenkins/workspace/tox-jessie/.tox/py27/bin:/usr/local/bin:/usr/bin:/bin:/usr/games', 'ZUUL_CHANGES': 'integration/config:master:refs/changes/71/243671/1', 'JOB_NAME': 'tox-jessie', 'ZUUL_PROJECT': 'integration/config', 'LANG': 'en_US.UTF-8', 'VIRTUAL_ENV': '/home/jenkins/workspace/tox-jessie/.tox/py27', 'SHELL': '/bin/bash', 'GIT_PREVIOUS_SUCCESSFUL_COMMIT': '12726579b2057bda6a1b715db605303c4c0c55cc', 'SHLVL': '2', 'NLSPATH': '/usr/dt/lib/nls/msg/%L/%N.cat', 'GIT_AUTHOR_EMAIL': 'Jenkins (Wikimedia)', 'HUDSON_HOME': '/var/lib/jenkins', 'ZUUL_CHANGE_IDS': '243671,1', 'NODE_LABELS': 'ci-jessie-wikimedia ci-jessie-wikimedia-3578', 'PYTHONHASHSEED': '3107003008', 'PY_COLORS': '1', 'BUILD_ID': '224', 'BUILD_NUMBER': '224', 'HUDSON_COOKIE': '37d5ed01-e16a-4180-a0f7-a06bf975cff0', 'JOB_URL': 'https://integration.wikimedia.org/ci/job/tox-jessie/', 'ZUUL_PATCHSET': '1', 'BASE_LOG_PATH': '71/243671/1', 'ZUUL_BRANCH': 'master', 'GIT_BRANCH': 'detached', 'GIT_COMMIT': '12726579b2057bda6a1b715db605303c4c0c55cc', 'GIT_AUTHOR_NAME': 'jenkins@gallium.wikimedia.org', 'ZUUL_URL': 'git://zuul.eqiad.wmnet', 'GIT_COMMITTER_NAME': 'jenkins@gallium.wikimedia.org', 'ZUUL_CHANGE': '243671', '_': '/usr/local/bin/tox', 'ZUUL_PIPELINE': 'gate-and-submit', 'ZUUL_UUID': 'acda23e9d6b04a2baed0b188a5e0e9cb', 'HUDSON_SERVER_COOKIE': '7e6d8c1ea8d4ef48', 'EXECUTOR_NUMBER': '0', 'NODE_NAME': 'ci-jessie-wikimedia-3578', 'LOG_PATH': '71/243671/1/gate-and-submit/tox-jessie/acda23e', 'PWD': '/home/jenkins/workspace/tox-jessie', 'GIT_COMMITTER_EMAIL': 'Jenkins (Wikimedia)', 'JENKINS_SERVER_COOKIE': '7e6d8c1ea8d4ef48', 'BUILD_DISPLAY_NAME': '#224', 'ZUUL_REF': 'refs/zuul/master/Z4fc4dd3411a44249866f4ea12293eaa3', 'XFILESEARCHPATH': '/usr/dt/app-defaults/%L/Dt', 'MAIL': '/var/mail/jenkins', 'SSH_CONNECTION': '208.80.154.135 62908 10.68.16.68 22', 'ZUUL_COMMIT': '12726579b2057bda6a1b715db605303c4c0c55cc', 'GIT_URL': 'git://zuul.eqiad.wmnet/integration/config'}
15:09:09 Collecting nose (from -r test-requirements.txt (line 1))
15:09:09   Downloading nose-1.3.7-py2-none-any.whl (154kB)
15:09:09 Collecting nose-timer (from -r test-requirements.txt (line 2))
15:09:09   Downloading nose-timer-0.5.0.tar.gz
15:09:09 Collecting PyYAML (from -r test-requirements.txt (line 3))
15:09:09   Downloading PyYAML-3.11.tar.gz (248kB)
15:09:09 Collecting zuul from git+https://gerrit.wikimedia.org/r/p/integration/zuul.git@debian/precise-wikimedia#egg=zuul (from -r test-requirements.txt (line 4))
15:09:09   Cloning https://gerrit.wikimedia.org/r/p/integration/zuul.git (to debian/precise-wikimedia) to /tmp/pip-build-wSiOCU/zuul
15:09:09 Collecting termcolor (from nose-timer->-r test-requirements.txt (line 2))
15:09:09   Downloading termcolor-1.1.0.tar.gz
15:09:09 Collecting pbr<1.0,>=0.5.21 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading pbr-0.11.0-py2.py3-none-any.whl (78kB)
15:09:09 Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from zuul->-r test-requirements.txt (line 4))
15:09:09 Collecting Paste (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading Paste-2.0.2-py2-none-any.whl (610kB)
15:09:09 Collecting WebOb<1.3,>=1.2.3 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading WebOb-1.2.3.tar.gz (191kB)
15:09:09 Collecting paramiko>=1.8.0 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading paramiko-1.15.3-py2.py3-none-any.whl (166kB)
15:09:09 Collecting GitPython>=0.3.2.1 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading GitPython-1.0.1.tar.gz (355kB)
15:09:09 Collecting lockfile>=0.8 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading lockfile-0.10.2-py2-none-any.whl
15:09:09 Collecting ordereddict (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading ordereddict-1.1.tar.gz
15:09:09 Collecting python-daemon>=2.0.4 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading python_daemon-2.0.6-py2.py3-none-any.whl (54kB)
15:09:09 Collecting extras (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading extras-0.0.3.tar.gz
15:09:09 Collecting statsd<3.0,>=1.0.0 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading statsd-2.1.2.tar.gz
15:09:09 Collecting voluptuous>=0.7 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading voluptuous-0.8.7.tar.gz
15:09:09 Collecting gear<1.0.0,>=0.5.4 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading gear-0.5.8-py2-none-any.whl
15:09:09 Collecting apscheduler<3.0,>=2.1.1 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading APScheduler-2.1.2-py2.py3-none-any.whl
15:09:09 Collecting PrettyTable<0.8,>=0.6 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading prettytable-0.7.2.tar.bz2
15:09:09 Collecting babel>=1.0 (from zuul->-r test-requirements.txt (line 4))
15:09:09   Downloading Babel-2.1.1-py2.py3-none-any.whl (3.6MB)
15:09:09 Exception:
15:09:09 Traceback (most recent call last):
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/basecommand.py", line 211, in main
15:09:09     status = self.run(options, args)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/commands/install.py", line 305, in run
15:09:09     wb.build(autobuilding=True)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/wheel.py", line 705, in build
15:09:09     self.requirement_set.prepare_files(self.finder)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/req/req_set.py", line 334, in prepare_files
15:09:09     functools.partial(self._prepare_file, finder))
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
15:09:09     more_reqs = handler(req_to_install)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/req/req_set.py", line 491, in _prepare_file
15:09:09     session=self.session)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/download.py", line 825, in unpack_url
15:09:09     session,
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/download.py", line 673, in unpack_http_url
15:09:09     from_path, content_type = _download_http_url(link, session, temp_dir)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/download.py", line 886, in _download_http_url
15:09:09     _download_url(resp, link, content_file)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/download.py", line 621, in _download_url
15:09:09     for chunk in progress_indicator(resp_read(4096), 4096):
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/utils/ui.py", line 133, in iter
15:09:09     for x in it:
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/download.py", line 586, in resp_read
15:09:09     decode_content=False):
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 307, in stream
15:09:09     data = self.read(amt=amt, decode_content=decode_content)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 243, in read
15:09:09     data = self._fp.read(amt)
15:09:09   File "/home/jenkins/workspace/tox-jessie/.tox/py27/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 49, in read
15:09:09     data = self.__fp.read(amt)
15:09:09   File "/usr/lib/python2.7/httplib.py", line 573, in read
15:09:09     s = self.fp.read(amt)
15:09:09   File "/usr/lib/python2.7/socket.py", line 380, in read
15:09:09     data = self._sock.recv(left)
15:09:09   File "/usr/lib/python2.7/ssl.py", line 714, in recv
15:09:09     return self.read(buflen)
15:09:09   File "/usr/lib/python2.7/ssl.py", line 608, in read
15:09:09     v = self._sslobj.read(len or 1024)
15:09:09 error: [Errno 104] Connection reset by peer
15:09:09 
15:09:09 ERROR: could not install deps [-rtest-requirements.txt]; v = InvocationError('/home/jenkins/workspace/tox-jessie/.tox/py27/bin/pip install -rtest-requirements.txt (see /home/jenkins/workspace/tox-jessie/.tox/py27/log/py27-1.log)', 2)
15:09:09 ___________________________________ summary ____________________________________
15:09:09   flake8: commands succeeded
15:09:09 ERROR:   py27: could not install deps [-rtest-requirements.txt]; v = InvocationError('/home/jenkins/workspace/tox-jessie/.tox/py27/bin/pip install -rtest-requirements.txt (see /home/jenkins/workspace/tox-jessie/.tox/py27/log/py27-1.log)', 2)

Related Objects

Event Timeline

JanZerebecki created this task.Oct 5 2015, 3:20 PM
JanZerebecki raised the priority of this task from to Needs Triage.
JanZerebecki updated the task description. (Show Details)
JanZerebecki added projects: Continuous-Integration-Scaling, Continuous-Integration-Infrastructure.
JanZerebecki subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2015, 3:20 PM
JanZerebecki renamed this task from tox-jessie job is failing randomly on integration/config with could not install deps to tox-jessie job is failing randomly on integration/config with could not install deps, because of throttling.Oct 6 2015, 2:56 PM
JanZerebecki updated the task description. (Show Details)
JanZerebecki set Security to None.
hashar subscribed.Oct 7 2015, 12:36 PM

From yesterday meeting, that is Pypi causing some connection issue or maybe the labs web proxy if we use it at all.

Related are:

By using our own cache, we would potentially mitigate reliability issue with Pypi.

hashar closed this task as Resolved.Dec 21 2015, 2:47 PM
hashar claimed this task.

Transient issue due to Pypi. Will be less of an issue whenever we have our own cache / mirror (T112560).

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 21 2015, 2:47 PM
greg added a project: Essential-Work.Jan 11 2016, 10:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL