CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Zabe
	May 21 2022, 7:58 PM

Description

Changes without a cross-repo dependency and which apply on master without a conflict fail CI with 'This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.' in quite a few repos.

Random example: https://gerrit.wikimedia.org/r/c/integration/config/+/794756

https://gerrit.wikimedia.org/r/c/mediawiki/libs/IDLeDOM/+/794688 looks like the first affected patch (May 21, 2022, 19:55:49 UTC+01:00)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		hashar	T309376 Zuul jenkins-bot user holding open SSH sessions
		Resolved		hashar	T308943 CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos

Event Timeline

Zabe created this task.May 21 2022, 7:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 21 2022, 7:58 PM

Mitar subscribed.May 21 2022, 8:02 PM

Same here: https://gerrit.wikimedia.org/r/794759 - my guess is that this is a problem with zuul-merger?

matmarex subscribed.May 21 2022, 8:45 PM

TheresNoTime subscribed.May 21 2022, 8:55 PM

zuul-merger is apparently running and the queue isn't too bad — someone with access to contint2001 please check the logs, as the last 3 patches in gerrit have all failed

Looking closer, this is affecting every patch — raising to UBN!

TheresNoTime updated the task description. (Show Details)May 21 2022, 9:39 PM

There are a lot of errors similar to this in /var/log/zuul/merger-debug.log:

2022-05-21 21:50:11,137 DEBUG zuul.Merger: Unable to merge {u'oldrev': None, u'newrev': None, u'refspec': u'refs/changes/33/793933/10', u'merge_mode': 2, u'connection_name': u'gerrit', u'number': u'793933', u'project': u'mediawiki/extensions/BlueSpiceUEModulePDF', u'url': u'ssh://jenkins-bot@gerrit.wikimedia.org:29418/mediawiki/extensions/BlueSpiceUEModulePDF', u'branch': u'REL1_35', u'patchset': 10, u'ref': u'Z2faf357cf2c14eb4ad0927c11e49959d'}
Traceback (most recent call last):
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 277, in _mergeChange
    commit = repo.merge(item['refspec'], 'resolve')
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 163, in merge
    self.fetch(ref)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 176, in fetch
    origin.fetch(ref)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 789, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 675, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)
GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch -v origin refs/changes/33/793933/10
  stderr: 'fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.'

Stang subscribed.May 21 2022, 9:57 PM

o/ checking

(https://gerrit.wikimedia.org/r/c/mediawiki/extensions/BlueSpiceUEModulePDF/+/793933/10 is that ^ change fwiw)

From the Gerrit log https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV

Max connection count for user jenkins-bot exceeded, rejecting new connection. currentSessionCount = 4, maxSessionCount = 4

+ ssh -p 29418 hashar@gerrit.wikimedia.org gerrit show-connections
de2baa80   jenkins-bot     contint2001.wikimedi
009fa654   jenkins-bot     contint2001.wikimedi
606d4235   jenkins-bot     contint2001.wikimedi

Daimona subscribed.May 21 2022, 10:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-21T22:10:51Z] <hashar> Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit # T308943

Should be good now after I have restarted Zuul (ssh contint2001.wikimedia.org sudo systemctl restart zuul) which cleared the idling ssh connection.

Just confirming that https://gerrit.wikimedia.org/r/c/integration/config/+/794756, which was previously failing due to this, has now passed — thanks @hashar! 🥳

Thanks for the confirmation @TheresNoTime !

TheresNoTime added a subscriber: DannyS712.May 22 2022, 1:28 AM

thcipriani awarded a token.May 22 2022, 1:39 AM

Stang unsubscribed.May 22 2022, 5:26 AM

TheresNoTime mentioned this in T309371: Gerrit: all patches are being reported as merge conflicts.May 27 2022, 4:17 AM

TheresNoTime mentioned this in T309376: Zuul jenkins-bot user holding open SSH sessions.May 27 2022, 6:41 AM

TheresNoTime added a parent task: T309376: Zuul jenkins-bot user holding open SSH sessions.

dancy reopened this task as Open.Oct 26 2022, 11:04 PM

Mentioned in SAL (#wikimedia-releng) [2022-10-26T23:12:12Z] <dancy> Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit #T308943

Dzahn subscribed.Oct 26 2022, 11:13 PM

Same thing happened today. Resolved by restarting zuul.

Michael mentioned this in T344238: CI "Merge Failed. because cross-repo dependencies" on CI jobs, even up-to-date ones.Aug 15 2023, 11:35 AM

	F35161335: image.png
	May 21 2022, 9:22 PM

CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of reposClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos
Closed, ResolvedPublic
Actions

Related Objects
Search...