Page MenuHomePhabricator

Document Zuul problems caused by force merge
Open, MediumPublic

Description

Question from @awight on https://www.mediawiki.org/wiki/Topic:V14dlv7nt5ne7gsd

I remember that there have been some deadlocks caused every time devs did a force merge, and allegedly there is still a type of slowness which happens after a force merge, but I'm not clear on the details. It would be great to discuss here (if this is the right place), or link to a Phabricator task which discusses the issue.

We should document it.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 17 2019, 4:36 PM

In short Zuul is supposed to handle all the merges and thus it misbehave whenever a change is force merged. A few things on top of my mind:

Once Zuul got a change merged, it would wait for the change to show up in the repository and hold. That is for the case when using Gerrit git replica and to wait for the commit to have been replicated. When another change is force merged, Zuul got confused since the tip of the branch does not match its expectation.

When a change got force merged but is already in gate-and-submit, when Zuul tries to submit it Gerrit will complain (the change is closed). Zuul then considers the action has failed and would remove the change from the queue assuming it failed. The changes behind ends up having all their jobs canceled and rescheduled.

And of course, changes being force merged tends to eventually break tests for the next builds. Sometime even for other repositories specially when some jobs is running across several repositories (mediawiki extensions dependencies for example).

greg assigned this task to hashar.Jun 17 2019, 4:55 PM
greg moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
greg triaged this task as Medium priority.Jun 17 2019, 6:51 PM
greg added a project: Documentation.