Page MenuHomePhabricator
Create Task
Maniphest T128569

Zuul deadlocks if unknown repo has activity in Gerrit
Closed, ResolvedPublic
Actions

Description

In the night of Friday to Saturday, Zuul got caught in a death loop that prevented it from processing changes. @JanZerebecki had to restart Zuul entirely.

It occurred again. The exception reported looks like:

2016-03-02 01:20:04,597 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 836, in run
    while pipeline.manager.processQueue():
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1456, in processQueue
    item, nnfi)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1414, in _processOneItem
    ready = self.prepareRef(item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1321, in prepareRef
    url = self.pipeline.source.getGitUrl(item.change.project)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py", line 500, in getGitUrl
    url = 'ssh://%s@%s:%s/%s' % (user, server, port, project.name)
AttributeError: 'NoneType' object has no attribute 'name'

It happens when the Zuul scheduler prepare a reference for a project. For some reason the project is missing a name which should really not happen :-}

It magically solved itself with last error being reported at 2016-03-02 08:32:19.

Details

SubjectRepoBranchLines +/-
2.5.0-8-gcbc7f62-wmf1precise1integration/zuuldebian/precise-wikimedia+11 -0
Test dependency to unknown repositoryintegration/configmaster+0 -0
Fix gerrit stalling because of an unkown projectintegration/zuuldebian/precise-wikimedia+150 -22
Fix gerrit stalling because of an unkown projectintegration/zuuldebian/jessie-wikimedia+150 -22
Fix gerrit stalling because of an unkown projectintegration/zuuldebian/trusty-wikimedia+150 -22
[maps/tilerator/deploy] define noopintegration/configmaster+6 -0
[analytics/wmde/scripts] define noopintegration/configmaster+6 -0
[mediawiki/event-schemas] Dummy definitionintegration/configmaster+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
hashar added a subscriber: thcipriani.Mar 2 2016, 10:14 AM

Seems the issue has been triggered by https://gerrit.wikimedia.org/r/#/c/274318/ which is a change against mediawiki/vagrant and with a few Depends-On headers for changes that got earlier:

https://gerrit.wikimedia.org/r/#/c/274312/I1f194cc318c18e7c292c5450ea011811de8df204mediawiki/event-schemas
https://gerrit.wikimedia.org/r/#/c/274313/Ibbf84cdf0c923a0a69a3d68226ffcf1c5c5a3361mediawiki/core

The change got created on 2016-03-02 00:08:59 UTC and has ChangeId I3bbc1e809a40a408c820a43f651c0624cf9675c5

The chain of events from the debug.log. At first event is received and dependencies received (by searching for change ID in other commit as well as looking for the Gerrit built-in dependency tree):

00:09:04,997 INFO zuul.Gerrit: Updating information for 274318,1
00:09:06,382 DEBUG zuul.Gerrit: Running query change:Ibbf84cdf0c923a0a69a3d68226ffcf1c5c5a3361 to find needed changes
00:09:06,554 DEBUG zuul.Gerrit: Running query change:I1f194cc318c18e7c292c5450ea011811de8df204 to find needed changes
00:09:07,414 DEBUG zuul.Gerrit: Getting commit-dependent change 274313,1
00:09:07,415 DEBUG zuul.Gerrit: Getting commit-dependent change 274312,1
00:09:07,415 DEBUG zuul.Gerrit: Running query message:I3bbc1e809a40a408c820a43f651c0624cf9675c5 to find changes needed-by

The trigger event is added and then processed by the scheduler pipelines. The item is properly added to the test pipeline:

00:09:15,753 INFO zuul.Scheduler: Adding mediawiki/vagrant, <Change 0x7f4980f458d0 274318,1> to <Pipeline test>
00:09:15,754 DEBUG zuul.IndependentPipelineManager: Considering adding change <Change 0x7f4980f458d0 274318,1>
00:09:15,757 DEBUG zuul.IndependentPipelineManager: Checking for changes needed by <Change 0x7f4980f458d0 274318,1>: 
00:09:15,757 DEBUG zuul.IndependentPipelineManager:   Change <Change 0x7f4980f458d0 274318,1> needs change <Change 0x7f4983f5b490 274313,1>: 
00:09:15,757 DEBUG zuul.IndependentPipelineManager:   Change <Change 0x7f4983f5b490 274313,1> is needed
00:09:15,758 DEBUG zuul.IndependentPipelineManager:   Change <Change 0x7f4980f458d0 274318,1> needs change <Change 0x2a08390 274312,1>: 
00:09:15,758 DEBUG zuul.IndependentPipelineManager:   Change <Change 0x2a08390 274312,1> is needed
00:09:15,758 DEBUG zuul.IndependentPipelineManager:   Changes [<Change 0x7f4983f5b490 274313,1>, <Change 0x2a08390 274312,1>] must be merged ahead of <Change 0x7f4980f458d0 274318,1>
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Considering adding change <Change 0x7f4983f5b490 274313,1>
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Checking for changes needed by <Change 0x7f4983f5b490 274313,1>: 
00:09:15,758 DEBUG zuul.IndependentPipelineManager:   No changes needed
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Adding change <Change 0x7f4983f5b490 274313,1> to queue <ChangeQueue test: mediawiki/vagrant>
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Considering adding change <Change 0x2a08390 274312,1>
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Checking for changes needed by <Change 0x2a08390 274312,1>: 
00:09:15,758 DEBUG zuul.IndependentPipelineManager:   No changes needed
00:09:15,758 DEBUG zuul.IndependentPipelineManager: Adding change <Change 0x2a08390 274312,1> to queue <ChangeQueue test: mediawiki/vagrant>

Then the scheduler report its pipeline status to statsd. To forge the metric names, it replaces / in Gerrit project names with dots .. Somehow one change (no clue which one) is missing the project name:

00:09:15,759 ERROR zuul.IndependentPipelineManager: Exception reporting pipeline stats
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1767, in reportStats
    project_name = item.change.project.name.replace('/', '.')
AttributeError: 'NoneType' object has no attribute 'name'

The vagrant change is being added to the test pipeline:

00:09:15,849 DEBUG zuul.IndependentPipelineManager: Adding change <Change 0x7f4980f458d0 274318,1> to queue <ChangeQueue test: mediawiki/vagrant>

When preparing the Zuul refs (a merge commit of patchset on tip of target branch) for the two dependent changes:

00:09:15,850 DEBUG zuul.IndependentPipelineManager: Preparing ref for: <Change 0x7f4983f5b490 274313,1>
00:09:15,850 DEBUG zuul.MergeClient: Submitting job <gear.Job 0x7f4981164d10 handle: None name: merger:merge unique: 596c2a9451e84083a4307c4fe004188a> with data
{'items': [
  {'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/13/274313/1',
   'merge_mode': 2, 'number': u'274313',
   'project': 'mediawiki/core',
   'url': 'ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/core',
   'branch': u'master',
   'patchset': u'1', 'ref': 'Z628016885c3843928c549ae5b84e330b'}]}

The dependent change 274313 definitely has a project. It is then time for the other dependent change:

00:09:15,852 DEBUG zuul.IndependentPipelineManager: Preparing ref for: <Change 0x2a08390 274312,1>
00:09:15,852 INFO zuul.IndependentPipelineManager: Change <Change 0x2a08390 274312,1> depends on changes [<Change 0x7f4983f5b490 274313,1>]
00:09:15,852 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 836, in run
    while pipeline.manager.processQueue():
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1456, in processQueue
    item, nnfi)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1414, in _processOneItem
    ready = self.prepareRef(item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1315, in prepareRef
    merger_items = map(self._makeMergerItem, all_items)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1288, in _makeMergerItem
    return dict(project=item.change.project.name,
AttributeError: 'NoneType' object has no attribute 'name'

So somehow when Zuul got the informations for 274312,1 it failed to acquire the project.

From there whenever the scheduler run handler process the changes it knows about, it keeps erroring out because of that change.

The mediawiki/core change 274313 got processed properly and had tests run for it:

23:29:06,298 INFO zuul.Gerrit: Updating information for 274313,1
23:29:16,096 DEBUG zuul.IndependentPipelineManager: Adding change <Change 0x7f4983f5b490 274313,1> to queue <ChangeQueue test: mediawiki>
23:29:16,097 DEBUG zuul.IndependentPipelineManager: Preparing ref for: <Change 0x7f4983f5b490 274313,1>
23:29:16,097 DEBUG zuul.MergeClient: Submitting job <gear.Job 0x7f4981147890 handle: None name: merger:merge unique: 28a45abfe1bd4316ad9a8d49f206e7a0> with data
{'items': [{
  'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/13/274313/1', 'merge_mode': 2,
  'number': u'274313',
  'project': 'mediawiki/core',
  'url': 'ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/core',
  'branch': u'master', 'patchset': u'1', 'ref': 'Z3fd08e960c424c24888cd4e58dfb6708'}]}

For the mediawiki/event-schemas change 274312 that is another story:

23:27:29,552 INFO zuul.Gerrit: Updating information for 274312,1
23:27:41,157 DEBUG zuul.Scheduler: Done adding trigger event: <TriggerEvent patchset-created mediawiki/event-schemas master 274312,1>
23:27:41,158 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent patchset-created mediawiki/event-schemas master 274312,1>

23:27:41,158 DEBUG zuul.Scheduler: Project mediawiki/event-schemas not found

23:27:41,158 DEBUG zuul.IndependentPipelineManager: Starting queue processor: test
23:27:41,158 DEBUG zuul.IndependentPipelineManager: Finished queue processor: test (changed: False)

The project is not found! The message is emitted when processing the event queue. It looks in the layout whether the project is defined and skip the event if not:

project = self.layout.projects.get(event.project_name)
if not project:
    self.log.debug("Project %s not found" % event.project_name)
    return

As a result the change is not enqueued and has not been added to the test pipeline (changed: False above).

So i have no idea why:

  • mediawiki/event-schemas is not found in the layout file
  • the change item ends up missing the project name

The whole issue got fixed by @thcipriani which restarted Zuul:

01:19	<thcipriani>	force restarting zuul because the queue is very stuck https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
01:20:22,265 INFO zuul.Server: Starting scheduler
hashar added a comment.Mar 2 2016, 10:18 AM

I did a recheck of https://gerrit.wikimedia.org/r/#/c/274312/1

2016-03-02 10:15:48,293 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added mediawiki/event-schemas master 274312,1>
2016-03-02 10:15:48,293 DEBUG zuul.Scheduler: Project mediawiki/event-schemas not found

And indeed that repository is not defined in the Zuul layout. That would cause a corner case issue that cause the change item to not be properly defined.

hashar claimed this task.Mar 2 2016, 10:18 AM
hashar triaged this task as Unbreak Now! priority.
gerritbot subscribed.Mar 2 2016, 10:23 AM

Change 274356 had a related patch set uploaded (by Hashar):
[mediawiki/event-schemas] Dummy definition

https://gerrit.wikimedia.org/r/274356

gerritbot added a project: Patch-For-Review.Mar 2 2016, 10:23 AM
gerritbot added a comment.Mar 2 2016, 10:25 AM

Change 274356 merged by jenkins-bot:
[mediawiki/event-schemas] Dummy definition

https://gerrit.wikimedia.org/r/274356

Stashbot added a comment.Mar 2 2016, 10:38 AM

Mentioned in SAL [2016-03-02T10:38:27Z] <hashar> Zuul should no more be caught in death loop due to Depends-On on an event-schemas change. Hole filled with https://gerrit.wikimedia.org/r/#/c/274356/ T128569

hashar added a comment.Mar 2 2016, 10:38 AM

Did a recheck on the change having the two dependent changes ( https://gerrit.wikimedia.org/r/#/c/274318/ ). Zuul properly handled the event-schema dependent change 274312

10:27:46,142 DEBUG zuul.IndependentPipelineManager: Preparing ref for: <Change 0x7fbe1e684810 274312,1>
10:27:46,145 DEBUG zuul.MergeClient: Submitting job <gear.Job 0x7fbe1eab4e10 handle: None name: merger:merge unique: e001931672d2405895025b93484a8065> with data
{'items': [
{'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/13/274313/1', 'merge_mode': 2, 'number': u'274313', 'project': 'mediawiki/core', 'url': 'ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/core', 'branch': u'master', 'patchset': u'1', 'ref': 'Za0f51fb06fa34ab9a41d61b68b93313b'},
{'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/12/274312/1', 'merge_mode': 2, 
'number': u'274312',                  # <----------
'project': 'mediawiki/event-schemas', # <----------
'url': 'ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/event-schemas', 'branch': u'master', 'patchset': u'1', 'ref': 'Z233c94126be94bf88109d3f4448e4310'},
{'oldrev': None, 'newrev': None, 'refspec': u'refs/changes/18/274318/1', 'merge_mode': 2, 'number': u'274318', 'project': 'mediawiki/vagrant', 'url': 'ssh://jenkins-bot@ytterbium.wikimedia.org:29418/mediawiki/vagrant', 'branch': u'master', 'patchset': u'1', 'ref': 'Zc225cfdedff4465f9573594cd0ef2a9b'}]}

Tests got run properly as expected.

hashar closed this task as Resolved.Mar 2 2016, 10:42 AM
hashar edited projects, added Essential-Work; removed Patch-For-Review.

The root cause was a change depending on a change made on a repository not known to Zuul. It causes the dependent change to not be fully known by Zuul and namely missing the project => death loop.

The proper fix is to have all repositories defined in Zuul layout.

Zuul assumes the project is available on a change and thus does not deal with error condition which should really not happen ™.

A similar issue occurred on Saturday, I have not investigated it but the stracktraces were identical. It was mentioned by @JanZerebecki during the CI meeting. I quickly looked at it and dismissed investigation (was late, assumed it is a weird corner case that would not happen again).

Stashbot added a comment.Mar 2 2016, 10:42 AM

Mentioned in SAL [2016-03-02T10:42:39Z] <hashar> Zuul should no more be caught in death loop due to Depends-On on an event-schemas change. Hole filled with https://gerrit.wikimedia.org/r/#/c/274356/ T128569

hashar mentioned this in T128462: JJB should support YAML axis .Mar 2 2016, 10:46 AM
Krinkle renamed this task from Zuul get caught in an error loop preventing it from processing changes to Zuul deadlocks if unknown repo has activity in Gerrit.Mar 14 2016, 5:01 PM
Krinkle reopened this task as Open.
Krinkle subscribed.

This was not resolved. It was hacked around.

Happening again:

2016-03-14 17:00:28,100 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 836, in run
    while pipeline.manager.processQueue():
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1456, in processQueue
    item, nnfi)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1414, in _processOneItem
    ready = self.prepareRef(item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1321, in prepareRef
    url = self.pipeline.source.getGitUrl(item.change.project)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py", line 500, in getGitUrl
    url = 'ssh://%s@%s:%s/%s' % (user, server, port, project.name)
AttributeError: 'NoneType' object has no attribute 'name'

Presumably triggered by https://gerrit.wikimedia.org/r/#/c/277264/.

Stashbot added a comment.Mar 14 2016, 5:03 PM

Mentioned in SAL [2016-03-14T17:02:59Z] <Krinkle> Doing full Zuul restart due to deadlock (T128569)

Paladox subscribed.Mar 14 2016, 5:03 PM

Another crash happended because of an unkown project which I have added here https://gerrit.wikimedia.org/r/277268

Paladox added a comment.Mar 14 2016, 5:08 PM

A bug needs to be filed upstream with high priority since this will affect everyone.

Krinkle added a comment.Mar 14 2016, 5:09 PM

@hashar Zuul didn't used to crash when Gerrit receives a commit to a repo not known to Zuul. This used to work fine. Can you confirm this is an upstream regression? Maybe it is already fixed in latest upstream?

Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Mar 14 2016, 5:09 PM
Paladox added a comment.Mar 14 2016, 5:16 PM

I think its fixed upstream here https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce

Paladox added a comment.Mar 14 2016, 5:23 PM

Looks like we either have to wait for them to release there next version which may be a while or we can update straight of the master branch. OpenStack uses it of the master branch so looks stable plus fixes the crash issue I hope.

hashar added a comment.Mar 14 2016, 8:07 PM
In T128569#2119019, @Krinkle wrote:

@hashar Zuul didn't used to crash when Gerrit receives a commit to a repo not known to Zuul. This used to work fine. Can you confirm this is an upstream regression? Maybe it is already fixed in latest upstream?

Zuul works fine when it receives from Gerrit an event for a repository not known to Zuul. But when there is a Depends-On set that points to a repo not know to Zuul it dies. So it is not a regression but a bug in the cross repository dependency management.

@Paladox wrote:

I think its fixed upstream here https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce
Looks like we either have to wait for them to release there next version which may be a while or we can update straight of the master branch. OpenStack uses it of the master branch so looks stable plus fixes the crash issue I hope.

Yup that upstream change looks correct at first glance. I will review it and it is not too far from our current version. Once happy, I will bump our Zuul version or cherry pick that patch then upgrade our Zuul. Good finding!

hashar added a project: Upstream.Mar 14 2016, 8:07 PM
hashar moved this task from Backlog to Patch merged upstream on the Upstream board.
Paladox added a comment.Mar 14 2016, 8:33 PM

Thanks.

Paladox added a comment.Mar 14 2016, 8:42 PM

We may want to include all changes in the master branch because I think it includes a fix for something performance wise with gerrit. Maybe it will fix the issue we have whereby it takes more then 5sec for a change to be recognised and the builds start.

Paladox added a comment.Mar 16 2016, 10:11 AM

@hashar and @Krinkle ive found an another fix that actually does fix it. https://github.com/openstack-infra/zuul/commit/07cc33c85c2979c9584d8473cb0a3d7934f146dd

It was fix a week after 2.1.0 update was published so not long. But we should still go up to the commit I linked to above to make sure it does not happened.

Paladox added a comment.Mar 16 2016, 10:19 AM

https://phabricator.wikimedia.org/rCIZU07cc33c85c2979c9584d8473cb0a3d7934f146dd

Krinkle unsubscribed.Mar 17 2016, 12:07 AM
Stashbot added a comment.Apr 26 2016, 8:07 PM

Mentioned in SAL [2016-04-26T20:07:31Z] <hashar> Killing Zuul entirely due to deadlock T128569

Restricted Application added subscribers: Southparkfan, Luke081515, TerraCodes, Urbanecm. · View Herald TranscriptApr 26 2016, 8:07 PM
hashar mentioned this in T134765: Zuul deadlock change_queue.isActionable(item) AttributeError: 'NoneType' object has no attribute 'isActionable'.May 9 2016, 2:55 PM
hashar merged a task: T134765: Zuul deadlock change_queue.isActionable(item) AttributeError: 'NoneType' object has no attribute 'isActionable'.
hashar added a subscriber: Zppix.
Addshore awarded a token.May 9 2016, 2:56 PM
Addshore subscribed.
gerritbot added a comment.May 9 2016, 2:58 PM

Change 287634 had a related patch set uploaded (by Hashar):
[analytics/wmde/scripts] define noop

https://gerrit.wikimedia.org/r/287634

gerritbot added a project: Patch-For-Review.May 9 2016, 2:58 PM
gerritbot added a comment.May 9 2016, 3:03 PM

Change 287634 merged by jenkins-bot:
[analytics/wmde/scripts] define noop

https://gerrit.wikimedia.org/r/287634

TerraCodes removed a project: Patch-For-Review.May 17 2016, 4:38 AM
gerritbot added a comment.May 27 2016, 5:52 PM

Change 291286 had a related patch set uploaded (by Luke081515):
[maps/tilerator/deploy] define noop

https://gerrit.wikimedia.org/r/291286

gerritbot added a project: Patch-For-Review.May 27 2016, 5:52 PM
gerritbot added a comment.May 27 2016, 5:57 PM

Change 291286 merged by Thcipriani:
[maps/tilerator/deploy] define noop

https://gerrit.wikimedia.org/r/291286

gerritbot added a comment.May 27 2016, 6:03 PM

Change 291289 had a related patch set uploaded (by Paladox):
Fix gerrit stalling because of an unkown project

https://gerrit.wikimedia.org/r/291289

gerritbot added a comment.May 27 2016, 6:04 PM

Change 291290 had a related patch set uploaded (by Paladox):
Fix gerrit stalling because of an unkown project

https://gerrit.wikimedia.org/r/291290

gerritbot added a comment.May 27 2016, 6:04 PM

Change 291291 had a related patch set uploaded (by Paladox):
Fix gerrit stalling because of an unkown project

https://gerrit.wikimedia.org/r/291291

Paladox lowered the priority of this task from Unbreak Now! to High.May 27 2016, 6:06 PM

Changing status to high since ive noticed this issue twice today requiring zuul to be restarted.

Paladox raised the priority of this task from High to Needs Triage.May 27 2016, 6:07 PM

Didn't realise it was on unbreak.

Paladox triaged this task as Unbreak Now! priority.May 27 2016, 6:19 PM
Luke081515 moved this task from Backlog to Bugs on the Zuul board.May 27 2016, 6:22 PM
hashar added a comment.May 30 2016, 12:51 PM

I have rebased our Zuul to include the patch 0deaaadac7143692961b9d28abee8cea570ff3ce and will attempt to upgrade.

Our new version is going to be zuul_2.1.0-95-g66c8e52-wmf1precise1

Stashbot added a comment.May 30 2016, 12:53 PM

Mentioned in SAL [2016-05-30T12:53:01Z] <hashar> Upgrading Zuul 1cc37f7..66c8e52 T128569

Stashbot added a comment.May 30 2016, 12:53 PM

Mentioned in SAL [2016-05-30T12:53:06Z] <hashar> Upgrading Zuul 1cc37f7..66c8e52 T128569

gerritbot added a comment.May 30 2016, 1:06 PM

Change 291730 had a related patch set uploaded (by Hashar):
Test dependency to unknown repository

https://gerrit.wikimedia.org/r/291730

gerritbot added a comment.May 30 2016, 1:12 PM

Change 291730 abandoned by Hashar:
Test dependency to unknown repository

https://gerrit.wikimedia.org/r/291730

hashar closed this task as Resolved.May 30 2016, 1:12 PM

Fixed by zuul_2.1.0-95-g66c8e52-wmf1precise1

hashar added a comment.May 30 2016, 1:15 PM

I have archived the Debian packages on rutherfordium.eqiad.wmnet:/home/hashar/public_html/debs/zuul_2.1.0-95-g66c8e52

Or: https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-95-g66c8e52/

gerritbot added a comment.May 30 2016, 1:20 PM

Change 291290 abandoned by Hashar:
Fix gerrit stalling because of an unkown project

Reason:
Upstream patches comes either from a bump of the 'upstream' branch or via a quilt package managed in the patch-queue/debian/precise-wikimedia branch.

https://gerrit.wikimedia.org/r/291290

gerritbot added a comment.May 30 2016, 1:20 PM

Change 291289 abandoned by Hashar:
Fix gerrit stalling because of an unkown project

https://gerrit.wikimedia.org/r/291289

gerritbot added a comment.May 30 2016, 1:20 PM

Change 291291 abandoned by Paladox:
Fix gerrit stalling because of an unkown project

https://gerrit.wikimedia.org/r/291291

Paladox moved this task from Next to Done on the Continuous-Integration-Infrastructure board.Jun 3 2016, 7:26 PM
Paladox removed a project: Patch-For-Review.Jun 3 2016, 8:30 PM
Paladox removed a subscriber: gerritbot.
hashar mentioned this in T129938: Circular dependencies break Zuul.Jul 26 2016, 12:46 PM
hashar reopened this task as Open.Jul 26 2016, 12:49 PM
hashar added a subscriber: Reedy.

Happened again with Zuul version: 2.1.0-151-g30a433b-wmf4precise1

In T129938#2495362, @Reedy wrote:

It was https://gerrit.wikimedia.org/r/#/c/300804

[23:43:56] <legoktm> yes that was the problem
[23:44:03] <legoktm> the vendor repo is not controlled by zuul

/var/log/zuul/error.log has a spam of:

2016-07-25 22:19:30,252 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 974, in run
    while pipeline.manager.processQueue():
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1610, in processQueue
    item, nnfi)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1568, in _processOneItem
    ready = self.prepareRef(item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1474, in prepareRef
    url = self.pipeline.source.getGitUrl(item.change.project)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/source/gerrit.py", line 354, in getGitUrl
    return self.connection.getGitUrl(project)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 433, in getGitUrl
    project.name)
AttributeError: 'NoneType' object has no attribute 'name'

It is supposedly fixed by rCIZU0deaaadac7143692961b9d28abee8cea570ff3ce which I have deployed end of May :(

greg subscribed.Jul 28 2016, 6:40 PM

Is this still UBN! ?

hashar lowered the priority of this task from Unbreak Now! to High.Jul 28 2016, 7:06 PM

I got Zuul upgraded on July 27th at 14:00 UTC: Zuul version: 2.1.0-391-gbc58ea3-wmf1precise1

Which is only a couple commits behind tip of master. rCIZU0deaaadac7143692961b9d28abee8cea570ff3ce is definitely included in it.

The trouble is digging in the log and figure out which corner case managed to escape the fix above.

Paladox added a comment.Aug 26 2016, 11:11 AM

@hashar looking at https://github.com/openstack-infra/zuul/commit/a8b90b38094587bbd82ffa5e4028aef1dfd02987 it seems there is another fix for this? Could we try it please?

Paladox added a comment.Aug 26 2016, 11:26 AM
This comment was removed by Paladox.
hashar mentioned this in T140894: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger).Aug 26 2016, 11:55 AM
gerritbot added a comment.Aug 27 2016, 12:19 PM

Change 307060 had a related patch set uploaded (by Hashar):
2.5.0-8-gcbc7f62-wmf1precise1

https://gerrit.wikimedia.org/r/307060

gerritbot added a project: Patch-For-Review.Aug 27 2016, 12:19 PM
gerritbot added a comment.Aug 27 2016, 12:26 PM

Change 307060 merged by Hashar:
2.5.0-8-gcbc7f62-wmf1precise1

https://gerrit.wikimedia.org/r/307060

hashar mentioned this in T144088: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1.Aug 27 2016, 12:29 PM
hashar created subtask T144088: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1.
Paladox removed a project: Patch-For-Review.Aug 27 2016, 4:16 PM
hashar added a comment.Aug 29 2016, 8:25 AM

I have upgraded Zuul server on gallium with zuul_2.5.0-8-gcbc7f62-wmf1precise1. It includes upstream patch @Paladox has mentioned:

a8b90b3 - Always create foreign projects if needed (4 days ago) <James E. Blair>

hashar closed this task as Resolved.Aug 29 2016, 8:31 AM

The main cause has been fixed by upstream commit: 0deaaadac7143692961b9d28abee8cea570ff3ce

With a follow up upstream to deal with a corner case issue in the changes cache which might still cause some projects to be None in a rare race condition: a8b90b38094587bbd82ffa5e4028aef1dfd02987

We have both deployed on our Zuul server as of zuul_2.5.0-8-gcbc7f62-wmf1precise1.

It must be fixed now. I am resolving this task, the sub task left open is "just" pending for the .deb package to be pushed to apt.wikimedia.org which is not a blocker.

hashar closed subtask T144088: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 as Resolved.Aug 29 2016, 10:51 AM
hashar reopened subtask T144088: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 as Open.
hashar closed subtask T144088: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 as Resolved.Oct 27 2016, 11:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL