Page MenuHomePhabricator

CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos
Closed, ResolvedPublic

Description

Changes without a cross-repo dependency and which apply on master without a conflict fail CI with 'This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.' in quite a few repos.

Random example: https://gerrit.wikimedia.org/r/c/integration/config/+/794756


Event Timeline

Same here: https://gerrit.wikimedia.org/r/794759 - my guess is that this is a problem with zuul-merger?

zuul-merger is apparently running and the queue isn't too bad — someone with access to contint2001 please check the logs, as the last 3 patches in gerrit have all failed

TheresNoTime triaged this task as Unbreak Now! priority.EditedMay 21 2022, 9:22 PM

Looking closer, this is affecting every patch — raising to UBN!

image.png (758×82 px, 6 KB)

There are a lot of errors similar to this in /var/log/zuul/merger-debug.log:

2022-05-21 21:50:11,137 DEBUG zuul.Merger: Unable to merge {u'oldrev': None, u'newrev': None, u'refspec': u'refs/changes/33/793933/10', u'merge_mode': 2, u'connection_name': u'gerrit', u'number': u'793933', u'project': u'mediawiki/extensions/BlueSpiceUEModulePDF', u'url': u'ssh://jenkins-bot@gerrit.wikimedia.org:29418/mediawiki/extensions/BlueSpiceUEModulePDF', u'branch': u'REL1_35', u'patchset': 10, u'ref': u'Z2faf357cf2c14eb4ad0927c11e49959d'}
Traceback (most recent call last):
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 277, in _mergeChange
    commit = repo.merge(item['refspec'], 'resolve')
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 163, in merge
    self.fetch(ref)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 176, in fetch
    origin.fetch(ref)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 789, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 675, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)
GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch -v origin refs/changes/33/793933/10
  stderr: 'fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.'

From the Gerrit log https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV

Max connection count for user jenkins-bot exceeded, rejecting new connection. currentSessionCount = 4, maxSessionCount = 4
+ ssh -p 29418 hashar@gerrit.wikimedia.org gerrit show-connections
de2baa80   jenkins-bot     contint2001.wikimedi
009fa654   jenkins-bot     contint2001.wikimedi
606d4235   jenkins-bot     contint2001.wikimedi

Mentioned in SAL (#wikimedia-operations) [2022-05-21T22:10:51Z] <hashar> Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit # T308943

Should be good now after I have restarted Zuul (ssh contint2001.wikimedia.org sudo systemctl restart zuul) which cleared the idling ssh connection.

Just confirming that https://gerrit.wikimedia.org/r/c/integration/config/+/794756, which was previously failing due to this, has now passed — thanks @hashar! 🥳