Page MenuHomePhabricator

CI "Merge Failed. because cross-repo dependencies" on CI jobs, even up-to-date ones
Closed, DuplicatePublic


CI failes with the following error message:

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset

even on changes that include no cross-repo dependencies and are up-to-date with the current master branch of their repo. See for example this Wikibase change: DNM CI test (I6bd9ed37)

Things that were tried so far

Prior similar incidents

Prior documentation

Event Timeline

Michael triaged this task as Unbreak Now! priority.EditedAug 15 2023, 11:16 AM

will add content in a second, but see

Thanks @Michael for filing this task. Restarting zuul helped a bit but only for a short period of time, after which the failed CI issue was back again.

Mentioned in SAL (#wikimedia-operations) [2023-08-15T12:02:48Z] <sukhe> sukhe@contint2002:~$ sudo systemctl restart zuul: T344238

Change 949026 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: raise maxConnectionsPerUser to 8

thcipriani claimed this task.

Following a zuul restart, I noticed there were still 3 connections to gerrit from zuul:

$ ssh -p 29418 gerrit show-connections | grep jenkins-bot
deadbeef   jenkins-bot     contint2002.wikimedi
2626b319   jenkins-bot     contint2002.wikimedi
125baedf   jenkins-bot     contint2002.wikimedi

I killed the oldest connection:

$ ssh -p 29418 gerrit close-connection deadbeef
closing connection deadbeef...

And I haven't seen an error in zuul-merger since.

My theory is the zuul-merger process is holding a broken connection to gerrit. Since we just restart the zuul-server, that connection still lingers. TODO for next time this happens is to check lsof for the zuul-merger process to confirm that's where the connection is coming from.

Updated documentation on MediaWiki.

Calling this done. Please re-open if you see this error again in the next few hours (otherwise file a new task)

Thank you @ssingh for the zuul restart. And thank you @Jelto and @eoghan for being on the contint servers looking into this already by the time I woke up <3.

Change 949026 merged by Jelto:

[operations/puppet@production] gerrit: raise maxConnectionsPerUser to 8