Page MenuHomePhabricator

Zuul cancels all changes when a change is manually merged
Closed, ResolvedPublic

Description

Given gate-and-submit has changes A <- B <- C each having jobs running

A user manually merges A, The jobs are still running in Zuul

When jobs are completed, Zuul submits the change with --verified +2

Gerrit reject the review since it tries to vote on a closed change

Zuul considers A failed to merge. It thus cancel jobs of B and C and rebuild the queue.

Result: waste of time.

Example with https://gerrit.wikimedia.org/r/c/458603 . /var/log/zuul/error.log has:

Exception: Gerrit error executing gerrit review
  --project mediawiki/extensions/UniversalLanguageSelector
  --message "Gate pipeline build succeeded."
  --verified 2 --submit 458603,2

The Gerrit list has a post (Gerrit 2.14: Cannot post votes or downgrade votes on closed changes) that exactly describes it. You are not supposed to change votes after a change is closed. And it is said:

Labels can be configured to allow for post-submit votes using the allowPostSubmit setting. However, as explained there, you can't cast a vote score lower than the submit-time score, by design.

Given we vote verified 2 and it is the highest score, that should be fine.

''It used to be possible to bypass the limitation with --force-message but that has been removed a few Gerrit versions ago''

Task

Find out in Gerrit how to configure allowPostSubmit for the verified label.

Event Timeline

hashar created this task.Sep 7 2018, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2018, 9:02 PM
hashar updated the task description. (Show Details)Sep 7 2018, 9:03 PM
hashar added a comment.Sep 7 2018, 9:21 PM

I have tried on https://gerrit.wikimedia.org/r/#/c/test/gerrit-ping/+/458903/ a merged change with Verified+2

$ gerrit review  --verified 2 458903,1
$

Trying to submit:

$ gerrit review --submit 458903,1
error: fatal: change is merged

fatal: one or more reviews failed; review output above

:-\

hashar updated the task description. (Show Details)Sep 7 2018, 9:23 PM

Change 458914 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

hashar added a comment.Sep 7 2018, 9:55 PM

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

greg added a subscriber: greg.Sep 10 2018, 3:45 PM

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

We probably want to decide T94322 before putting too much more effort in forking Zuul in this way.

Change 470845 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

Change 458914 merged by Hashar:
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

Change 470845 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

greg assigned this task to hashar.Nov 21 2018, 12:12 AM
hashar moved this task from Backlog to Bugs on the Zuul board.Nov 26 2018, 10:54 PM
Paladox moved this task from Bugs & stuff to Local hacks on the Gerrit board.Dec 14 2018, 9:15 PM
hashar renamed this task from Zuul cancel all changes when a change is manually merged to Zuul cancels all changes when a change is manually merged.Dec 18 2018, 4:10 PM
hashar added a comment.Apr 4 2019, 8:44 PM

I have missed upgrading Zuul but should do it next monday (T208426).

I have upgraded Zuul to 2.5.1-wmf.6 (T208426). This is now pending verification which I guess can be done by looking at the Zuul error.log.

The patch has some issues though:

2019-04-11 00:55:50,226 DEBUG zuul.reporter.gerrit.Reporter: Report change <Change 0x7f2e586e6390 502894,5>, params {'verified': -1}, message: Gate pipeline build failed.

- quibble-vendor-mysql-hhvm-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/44406/console : FAILURE in 15m 47s
- quibble-vendor-mysql-php70-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/19662/console : FAILURE in 6m 45s
- quibble-vendor-mysql-php71-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php71-docker/8970/console : FAILURE in 5m 45s
- quibble-vendor-mysql-php72-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/8762/console : FAILURE in 6m 25s
- mwgate-npm-node-6-docker https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/97254/console : SUCCESS in 1m 16s
- mwext-php70-phan-seccheck-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker/49211/console : SUCCESS in 4m 05s
- mwext-php70-phan-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/25981/console : SUCCESS in 3m 12s
2019-04-11 00:55:50,299 ERROR zuul.DependentPipelineManager: Exception while reporting:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1768, in _reportItem
    ret = self.sendReport(actions, self.pipeline.source, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1293, in sendReport
    ret = reporter.report(source, self.pipeline, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/reporter/gerrit.py", line 39, in report
    message, self.reporter_config)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 287, in review
    if 'submit' not in action and 'change is merged' in err:
UnboundLocalError: local variable 'err' referenced before assignment

My code logic was wrong. The intent is to ignore error when trying to submit a patch that is already merged by returning early. But my patch had 'submit' not in action which does the opposite.

Changing to:

if 'submit' in action and err and 'change is merged' in err:

Change 502984 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

Change 502984 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

greg triaged this task as Normal priority.Apr 16 2019, 12:13 AM
hashar closed this task as Resolved.Apr 17 2019, 12:58 PM

Should be good now!