CI "Merge Failed. because cross-repo dependencies" on CI jobs, even up-to-date ones
Closed, DuplicatePublic
Actions

Assigned To

Authored By

	Michael
	Aug 15 2023, 11:16 AM

Description

CI failes with the following error message:

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset

even on changes that include no cross-repo dependencies and are up-to-date with the current master branch of their repo. See for example this Wikibase change: DNM CI test (I6bd9ed37)

Things that were tried so far

Fetching puppet on zuul https://sal.toolforge.org/log/7szV-IkBhuQtenzvIe8C
Restarting zuul: https://sal.toolforge.org/log/D83v-IkBhuQtenzvGyJz
- helped for a very short time, errors quickly started up again

Prior similar incidents

In May 2022, there was T309371 and T308943 for which T309376: Zuul jenkins-bot user holding open SSH sessions was identified as the root cause.

Prior documentation

https://www.mediawiki.org/wiki/Continuous_integration/Zuul#All_Gerrit_patches_complain_of_merge_conflicts (has only a very little bit, pointing to the above tasks)

Details

	Subject	Repo	Branch	Lines +/-
	gerrit: raise maxConnectionsPerUser to 8	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects

Mentioned Here: T308943: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos
T309371: Gerrit: all patches are being reported as merge conflicts
T309376: Zuul jenkins-bot user holding open SSH sessions

Event Timeline

Michael created this task.Aug 15 2023, 11:16 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 15 2023, 11:16 AM

will add content in a second, but see https://gerrit.wikimedia.org/r/q/status:open

TheresNoTime subscribed.Aug 15 2023, 11:17 AM

Michael updated the task description. (Show Details)Aug 15 2023, 11:19 AM

Michael added a subscriber: Jakob_WMDE.

RhinosF1 subscribed.Aug 15 2023, 11:23 AM

Michael updated the task description. (Show Details)Aug 15 2023, 11:35 AM

Daimona subscribed.Aug 15 2023, 11:43 AM

Michael added projects: Continuous-Integration-Infrastructure, Release-Engineering-Team.Aug 15 2023, 11:44 AM

Michael added subscribers: Legoktm, bd808, thcipriani, dcaro.Aug 15 2023, 11:49 AM

Umherirrender added a subscriber: hashar.Aug 15 2023, 11:52 AM

Michael updated the task description. (Show Details)Aug 15 2023, 11:54 AM

Thanks @Michael for filing this task. Restarting zuul helped a bit but only for a short period of time, after which the failed CI issue was back again.

Michael updated the task description. (Show Details)Aug 15 2023, 11:57 AM

eoghan subscribed.Aug 15 2023, 11:58 AM

Jelto subscribed.Aug 15 2023, 11:59 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-15T12:02:48Z] <sukhe> sukhe@contint2002:~$ sudo systemctl restart zuul: T344238

Fabfur subscribed.Aug 15 2023, 12:08 PM

Change 949026 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: raise maxConnectionsPerUser to 8

https://gerrit.wikimedia.org/r/949026

gerritbot added a project: Patch-For-Review.Aug 15 2023, 12:10 PM

Following a zuul restart, I noticed there were still 3 connections to gerrit from zuul:

$ ssh gerrit.wikimedia.org -p 29418 gerrit show-connections | grep jenkins-bot
deadbeef   jenkins-bot     contint2002.wikimedi
2626b319   jenkins-bot     contint2002.wikimedi
125baedf   jenkins-bot     contint2002.wikimedi

I killed the oldest connection:

$ ssh gerrit.wikimedia.org -p 29418 gerrit close-connection deadbeef
closing connection deadbeef...

And I haven't seen an error in zuul-merger since.

My theory is the zuul-merger process is holding a broken connection to gerrit. Since we just restart the zuul-server, that connection still lingers. TODO for next time this happens is to check lsof for the zuul-merger process to confirm that's where the connection is coming from.

Updated documentation on MediaWiki.

Calling this done. Please re-open if you see this error again in the next few hours (otherwise file a new task)

Thank you @ssingh for the zuul restart. And thank you @Jelto and @eoghan for being on the contint servers looking into this already by the time I woke up <3.

hashar closed this task as a duplicate of T309376: Zuul jenkins-bot user holding open SSH sessions.Aug 21 2023, 3:03 PM

Change 949026 merged by Jelto: