zuul-merger fails when repository names overlaps
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	hashar
	Feb 10 2017, 5:31 PM

Description

When populating new zuul-merger, some merge operations will eventually fail for example:

GitCommandError: 'git clone -v ssh://jenkins-bot@gerrit.wikimedia.org:29418/operations/puppet /srv/zuul/git/operations/puppet' returned with exit code 128
stderr: 'fatal: destination path '/srv/zuul/git/operations/puppet' already exists and is not an empty directory.

In the above case, the repository operations/puppet/mariadb had a merge request handled. That creates the directory /srv/zuul/git/operations/puppet.

Later when a merge request is handled for operations/puppet git clone fails due to the path already existing.

Workaround

delete the directory entirely and manually clone as zuul user or recheck till a merge job runs on that host

better:

From T138455#2401076:

ssh scandium.eqiad.wmnet
sudo -H -u zuul bash -l
cd /srv/ssd/zuul/git/operations/software
git init .
git remote add origin ssh://jenkins-bot@gerrit.wikimedia.org:29418/operations/software
git remote set-head origin --auto

Fix up

zuul-merger should not just git-clone but be smarter and gracefully handle a directory that already exists.

Failure to set the symbolic ref for origin ( git remote set-head origin --auto ) causes:

2017-02-17 12:48:06,304 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/operations/software
2017-02-17 12:48:06,305 DEBUG zuul.Repo: Updating repository /srv/zuul/git/operations/software
2017-02-17 12:48:07,192 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f69dc6c57d0>
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 322, in _mergeItem
    repo.reset()
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 110, in reset
    repo.head.reference = origin.refs['HEAD']
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 706, in __getitem__
    raise IndexError("No item found with id %r" % (self._prefix + index))
IndexError: No item found with id u'origin/HEAD'

Related Objects
Search...

Status	Assigned	Task
Open	None	T291916 Tracking task for Bullseye migrations in production
In Progress	Dzahn	T327068 Bullseye upgrade for remaining Collab hosts
Resolved	Daimona	T334492 Create a new phan config file to make usage for libraries easier
Resolved	hashar	T334954 Raise version of PHP on integration.wikimedia.org from 7.3 to 7.4+
Open	None	T359795 Switch Jenkins instances from Java 11 to Java 17
In Progress	Dzahn	T334517 upgrade contint servers to bullseye
Declined	None	T157818 zuul-merger fails when repository names overlaps
Resolved	hashar	T158406 operations/software repo commits always getting -1 from Jenkins-bot

Event Timeline

hashar created this task.Feb 10 2017, 5:31 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2017, 5:31 PM

hashar mentioned this in T138455: operations/software.git commits failing as unable to merge?.Feb 10 2017, 5:32 PM

I thought ytterbium.wikimedia.org is no more? I thought that's cobalt now?

Good catch. That is a copy paste from the old setup, we now use gerrit.wikimedia.org

Did a basic attempt at https://review.openstack.org/#/c/432477/ . But really I dont think we can easily mimic git clone :/

hashar mentioned this in T158406: operations/software repo commits always getting -1 from Jenkins-bot.Feb 17 2017, 11:28 AM

hashar added a subtask: T158406: operations/software repo commits always getting -1 from Jenkins-bot.

hashar updated the task description. (Show Details)Feb 17 2017, 12:49 PM

hashar closed subtask T158406: operations/software repo commits always getting -1 from Jenkins-bot as Resolved.Feb 17 2017, 12:53 PM

Mentioned in SAL (#wikimedia-releng) [2017-04-04T21:26:47Z] <hashar> contint2001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy due to T157818

Mentioned in SAL (#wikimedia-releng) [2017-04-04T21:29:17Z] <hashar> contint1001 : rm -fR /srv/zuul/git/mediawiki/services/graphoid/deploy due to T157818

hashar triaged this task as Low priority.Jun 16 2017, 10:55 AM

hashar mentioned this in T196615: Add 'noop' CI to super projects mediawiki/extensions and mediawiki/skins.Dec 31 2018, 3:24 PM

Mentioned in SAL (#wikimedia-releng) [2019-07-09T15:29:39Z] <thcipriani> contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to T157818

Mentioned in SAL (#wikimedia-releng) [2019-07-10T13:30:20Z] <hashar> on zuul-merger: rm /srv/zuul/git/operations/software/gerrit due to T157818 # T189549

Stashbot mentioned this in T189549: Setup CI for operations/software/gerrit.Jul 10 2019, 1:30 PM

hashar mentioned this in T261996: labs/libraryupgrader patches are regularly failing CI with "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository" except they're up to date.Sep 4 2020, 8:54 AM

That got addressed upstream with https://review.opendev.org/c/zuul/zuul/+/787451 Support overlapping repos and a flat workspace scheme It might theoretically be ported back to our version but I don't think we should bother. A workaround is to prepopulate the zuul-merge repositories with all Gerrit hosted repositories so we can control the namespace clashes. But that would surface whenever a colliding repository is created later on so not ideal.

Given it is a rare occurrence on the zuul-merger have been warmed up and I have documented the workaround in the task description, I am declining it.

When bringing a new zuul-merger instance I went to populate all active code git repositories with:

gerrit ls-projects --type CODE --state ACTIVE | \
  xargs -I{} -n1 bash -c '
    mkdir -p "$(dirname {})"
    git clone "ssh://jenkins-bot@gerrit.wikimedia.org:29418/{}" \
        "/srv/zuul/git{}"

The only clashes are for:

wikimedia/fundraising/crm/drupal (since wikimedia/fundraising/crm has a data directory)
analytics/reportcard/data (since analytics/reportcard has a data directory)

Dzahn added a parent task: T334517: upgrade contint servers to bullseye.Wed, Apr 17, 9:08 PM

zuul-merger fails when repository names overlapsClosed, DeclinedPublicActions