zuul-merger fails when repository names overlaps
Open, LowPublic

Description

When populating new zuul-merger, some merge operations will eventually fail for example:

GitCommandError: 'git clone -v ssh://jenkins-bot@gerrit.wikimedia.org:29418/operations/puppet /srv/zuul/git/operations/puppet' returned with exit code 128
stderr: 'fatal: destination path '/srv/zuul/git/operations/puppet' already exists and is not an empty directory.

In the above case, the repository operations/puppet/mariadb had a merge request handled. That creates the directory /srv/zuul/git/operations/puppet.

Later when a merge request is handled for operations/puppet git clone fails due to the path already existing.

Workaround

  1. delete the directory entirely and manually clone as zuul user or recheck till a merge job runs on that host
  1. better:

From T138455#2401076:

ssh scandium.eqiad.wmnet
sudo -H -u zuul bash -l
cd /srv/ssd/zuul/git/operations/software
git init .
git remote add origin ssh://jenkins-bot@gerrit.wikimedia.org:29418/operations/software
git remote set-head origin --auto

Fix up

zuul-merger should not just git-clone but be smarter and gracefully handle a directory that already exists.

Failure to set the symbolic ref for origin ( git remote set-head origin --auto ) causes:

2017-02-17 12:48:06,304 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/operations/software
2017-02-17 12:48:06,305 DEBUG zuul.Repo: Updating repository /srv/zuul/git/operations/software
2017-02-17 12:48:07,192 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f69dc6c57d0>
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 322, in _mergeItem
    repo.reset()
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 110, in reset
    repo.head.reference = origin.refs['HEAD']
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 706, in __getitem__
    raise IndexError("No item found with id %r" % (self._prefix + index))
IndexError: No item found with id u'origin/HEAD'
hashar created this task.Feb 10 2017, 5:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2017, 5:31 PM
Paladox moved this task from Backlog to Bugs on the Zuul board.Feb 10 2017, 6:01 PM
Paladox added a subscriber: Paladox.

I thought ytterbium.wikimedia.org is no more? I thought that's cobalt now?

hashar updated the task description. (Show Details)Feb 10 2017, 8:18 PM

Good catch. That is a copy paste from the old setup, we now use gerrit.wikimedia.org

Did a basic attempt at https://review.openstack.org/#/c/432477/ . But really I dont think we can easily mimic git clone :/

hashar updated the task description. (Show Details)Feb 17 2017, 12:49 PM

Mentioned in SAL (#wikimedia-releng) [2017-04-04T21:26:47Z] <hashar> contint2001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy due to T157818

Mentioned in SAL (#wikimedia-releng) [2017-04-04T21:29:17Z] <hashar> contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy due to T157818

hashar triaged this task as Low priority.Jun 16 2017, 10:55 AM