Change force merged cause a deadlock in Zuul gate-and-submit pipeline
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	hashar
	Mar 24 2015, 9:35 PM

Description

We have noticed yesterday a deadlock in Zuul. When a change is in the gate-and-submit but get force merged by someone in Gerrit, Zuul sometime ends up sleeping up to 5 minutes. Meanwhile all changes behind in the queues are obviously blocked.

What is ever more annoying is that Zuul consider the change hasn't entered the repository and thus consider it to be a failure. It thus retrigger the whole queue. So given six patches +2ed and force merged, it takes 6 * 5 = 30 minutes for Zuul to resume to normal operations.

A stacktrace:

Thread: 140156753389312
  File "/usr/lib/python2.7/threading.py", line 524, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "zuul/scheduler.py", line 784, in run
    while pipeline.manager.processQueue():
  File "zuul/scheduler.py", line 1358, in processQueue
    item, nnfi, ready_ahead)
  File "zuul/scheduler.py", line 1330, in _processOneItem
    self.reportItem(item)
  File "zuul/scheduler.py", line 1421, in reportItem
    item.change.branch)
  File "zuul/trigger/gerrit.py", line 224, in isMerged
    if self.waitForRefSha(change.project, ref, change._ref_sha):
  File "zuul/trigger/gerrit.py", line 203, in waitForRefSha
    time.sleep(self.replication_retry_interval)

Example changes: 198885,1 and 198950,1

The questions are: why cant it find the sha1 commit since the change has actually been merged? Maybe it is missing a git remote update.

That also prevent the other queues from being run :(

Details

Subject	Repo	Branch	Lines +/-
WMF: reduce Gerrit replication delay from 300 to 15 secs	integration/zuul	patch-queue/debian/precise-wikimedia	+1 -1
Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia	integration/zuul	debian/jessie-wikimedia	+4 -7
2.1.0-391-gbc58ea3-wmf2precise1	integration/zuul	debian/precise-wikimedia	+35 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Declined		Krinkle	T93812 Change force merged cause a deadlock in Zuul gate-and-submit pipeline
		Resolved		hashar	T94322 Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit

Event Timeline

hashar created this task.Mar 24 2015, 9:35 PM

hashar raised the priority of this task from to Needs Triage.

hashar updated the task description. (Show Details)

hashar added projects: Continuous-Integration-Infrastructure, Zuul.

hashar added subscribers: hashar, Legoktm.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2015, 9:35 PM

hashar updated the task description. (Show Details)Mar 24 2015, 9:35 PM

hashar set Security to None.

I have looked at the debug log for change 198885,1

2015-03-24 21:30:19,442 DEBUG zuul.Gerrit: Checking if change <Change 0x7f78adba01d0 198885,1> is merged
2015-03-24 21:30:19,923 DEBUG zuul.Gerrit: Waiting for <Change 0x7f78adba01d0 198885,1> to appear in git repo
2015-03-24 21:35:23,663 DEBUG zuul.Gerrit: Change <Change 0x7f78adba01d0 198885,1> did not appear in the git repo

The code is zuul/trigger/gerrit.py class Gerrit, but I have no idea why it can not find the reference in the Gerrit repo since it has been merged. It does retry every 5 seconds apparently.

The method:

def waitForRefSha(self, project, ref, old_sha=''):
    # Wait for the ref to show up in the repo
    start = time.time()
    while time.time() - start < self.replication_timeout:
        sha = self.getRefSha(project.name, ref)
        if old_sha != sha:
            return True
        time.sleep(self.replication_retry_interval)
    return False

And it is invoked by:

def isMerged(self, change, head=None):
...
        ref = 'refs/heads/' + change.branch
        self.log.debug("Waiting for %s to appear in git repo" % (change))
        if self.waitForRefSha(change.project, ref, change._ref_sha):
            self.log.debug("Change %s is in the git repo" %
                           (change))
            return True
        self.log.debug("Change %s did not appear in the git repo" %
                       (change))
        return False

Thus waitForRefSha() is invoked with the tip of the branch (refs/heads/master) and the change reference. It will succeed when they are different which is not the case since the change just got merged.

Seems some parameters are inverted as well. Will have to look at it again when rested.

Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Mar 25 2015, 3:29 AM

Krinkle mentioned this in T94322: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit.Mar 29 2015, 12:17 AM

Krinkle triaged this task as High priority.Mar 29 2015, 7:42 AM

Krinkle added a subtask: T94322: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit.

Upstream wontfix.

If people are allowed to self-merge on a project, the project can't use Zuul. The two are mutually exclusive. Zero tolerance.

Krinkle added a project: Upstream.Apr 14 2015, 11:30 PM

Krenair subscribed.Apr 14 2015, 11:30 PM

Is there a link for an upstream ticket?

That was from a discussion on #openstack-infra http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-04-14.log

<jeblair> below is the original author of Zuul and one of the principal developers.

<Krinkle> Hm.. I'm seeing some strange behaviour in our Zuul instance. The gate had about a dozen things queued up in a dependent pipeline. Everything was passing. The top one was the slowest so everythign behind it was already 'success' but waiting. Then after the top one succeeded as well and completed, eveything behind it suddenly went from 'sucess' back to 'queued' and started over.
<Krinkle> jeblair: The change in question was merged directly by another user.
<jeblair> Krinkle: zuul expects to be the only thing with permissions to merge (gerrit "submit") changes

<Krinkle> jeblair: Some people bypass Jenkins for backports. It's a bad practice that evolved because Zuul automatically combines alll changes regardless of repo or branch in the same queue due to the simplification of jobs.

Krinkle moved this task from Backlog to Bugs on the Zuul board.Apr 15 2015, 10:36 PM

Krinkle moved this task from Next to Done on the Continuous-Integration-Infrastructure board.Apr 16 2015, 3:13 AM

jayvdb subscribed.Jun 18 2015, 2:09 PM

hashar closed subtask T94322: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit as Declined.Jun 22 2015, 1:01 PM

Legoktm mentioned this in T89263: php-composer-validate job should not fail if a composer.json file is removed from the repository.Jun 24 2015, 5:38 AM

hashar mentioned this in T114165: Job wikimedia-fundraising-civicrm fails on branch civi-4.6.9-deploy causing force merges that breaks Zuul scheduler.Sep 29 2015, 7:46 PM

Besides the deadlock, Krinkle also mentioned on IRC that for future jobs using that repo from zuul it won't contain the merged change unless something made zuul fetch updates.

Legoktm mentioned this in T117062: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock of Zuul).Oct 29 2015, 5:24 PM

hashar mentioned this in T126274: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch.Feb 9 2016, 1:17 AM

Change 301768 had a related patch set uploaded (by Hashar):
WMF: reduce Gerrit replication delay from 300 to 15 secs

https://gerrit.wikimedia.org/r/301768

Change 301768 merged by Hashar:
WMF: reduce Gerrit replication delay from 300 to 15 secs

https://gerrit.wikimedia.org/r/301768

Change 301769 had a related patch set uploaded (by Hashar):
2.1.0-391-gbc58ea3-wmf2precise1

https://gerrit.wikimedia.org/r/301769

Change 301774 had a related patch set uploaded (by Hashar):
2.1.0-391-gbc58ea3-wmf2precise1

https://gerrit.wikimedia.org/r/301774

Mentioned in SAL [2016-07-29T09:38:54Z] <hashar> Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1

Mentioned in SAL [2016-07-29T09:38:57Z] <hashar> Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1