Page MenuHomePhabricator

Zuul Clone frequently fails in php70-phan-docker job
Closed, ResolvedPublic

Description

Last week and today, I've seen the php70-phan-docker job fail numerous times. Each time with the same issues:

+ zuul-cloner --version
22:38:53 Zuul version: 2.5.0-8-gcbc7f62-wmf4jessie1
22:38:53 + zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace /src --cache-dir /srv/git https://gerrit.wikimedia.org/r/p mediawiki/core
22:38:54 INFO:zuul.Cloner:Loaded map containing 4 rules
22:38:54 INFO:zuul.CloneMapper:Workspace path set to: /src
22:38:54 INFO:zuul.CloneMapper:Mapping projects to workspace...
22:38:54 INFO:zuul.CloneMapper:  mediawiki/core -> /src
22:38:54 DEBUG:zuul.CloneMapper:Checking overlap in destination directories...
22:38:54 INFO:zuul.CloneMapper:Expansion completed.
22:38:54 INFO:zuul.Cloner:Preparing 1 repositories
22:38:54 INFO:zuul.Cloner:Creating repo mediawiki/core from cache /srv/git/mediawiki/core.git
22:38:54 DEBUG:git.cmd:AutoInterrupt wait stderr: "fatal: destination path '/src' already exists and is not an empty directory.\n"
22:38:54 Traceback (most recent call last):
22:38:54   File "/usr/bin/zuul-cloner", line 10, in <module>
22:38:54     sys.exit(main())
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/cloner.py", line 167, in main
22:38:54     cloner.main()
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/cloner.py", line 162, in main
22:38:54     cloner.execute()
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/lib/cloner.py", line 75, in execute
22:38:54     self.prepareRepo(project, dest)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/lib/cloner.py", line 160, in prepareRepo
22:38:54     repo = self.cloneUpstream(project, dest)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/lib/cloner.py", line 100, in cloneUpstream
22:38:54     new_repo = git.Repo.clone_from(repo_cache, dest)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/repo/base.py", line 966, in clone_from
22:38:54     return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/repo/base.py", line 912, in _clone
22:38:54     finalize_process(proc, stderr=stderr)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 155, in finalize_process
22:38:54     proc.wait(**kwargs)
22:38:54   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/cmd.py", line 335, in wait
22:38:54     raise GitCommandError(self.args, status, errstr)
22:38:54 git.exc.GitCommandError: 'git clone -v /srv/git/mediawiki/core.git /src' returned with exit code 128
22:38:54 stderr: 'fatal: destination path '/src' already exists and is not an empty directory.
22:38:54 '

This one is from https://integration.wikimedia.org/ci/job/mediawiki-core-php70-phan-docker/6335/console for https://gerrit.wikimedia.org/r/#/c/416592/

A recheck usually fixes it, so it isn't deterministic, but this shouldn't happen as often :)

Event Timeline

Recent examples from that log:

4 days ago:
https://integration.wikimedia.org/ci/job/mediawiki-core-php70-phan-docker/8912/console

Building remotely on integration-slave-docker-1004 (DebianJessieDocker m4executor) in workspace /srv/jenkins-workspace/workspace/mediawiki-core-php70-phan-docker
..
+ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace /src --cache-dir /srv/git https://gerrit.wikimedia.org/r/p mediawiki/core
..
INFO:zuul.Cloner:Creating repo mediawiki/core from cache /srv/git/mediawiki/core.git
DEBUG:git.cmd:AutoInterrupt wait stderr: "fatal: destination path '/src' already exists and is not an empty directory.\n"

5 days ago:
https://integration.wikimedia.org/ci/job/mediawiki-core-php70-phan-docker/8865/console

Building remotely on integration-slave-docker-1004 (DebianJessieDocker m4executor) in workspace /srv/jenkins-workspace/workspace/mediawiki-core-php70-phan-docker
..
+ zuul-cloner --color --verbose --map /srv/deployment/integration/slave-scripts/etc/zuul-clonemap.yaml --workspace /src --cache-dir /srv/git https://gerrit.wikimedia.org/r/p mediawiki/core
..
DEBUG:git.cmd:AutoInterrupt wait stderr: "fatal: destination path '/src' already exists and is not an empty directory.\n"

This might be a symptom of the issue described at T179963, which is that stuff from unrelated previous builds may've been left behind.

hashar claimed this task.
hashar subscribed.

I have looked at the build log and the only match I found is August 20 2018 build https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker-non-voting/1189/ I have refactored the jobs the week after and it is no more an issue.