Page MenuHomePhabricator

Zuul cancels all changes when a change is manually merged
Open, NormalPublic

Description

Given gate-and-submit has changes A <- B <- C each having jobs running

A user manually merges A, The jobs are still running in Zuul

When jobs are completed, Zuul submits the change with --verified +2

Gerrit reject the review since it tries to vote on a closed change

Zuul considers A failed to merge. It thus cancel jobs of B and C and rebuild the queue.

Result: waste of time.

Example with https://gerrit.wikimedia.org/r/c/458603 . /var/log/zuul/error.log has:

Exception: Gerrit error executing gerrit review
  --project mediawiki/extensions/UniversalLanguageSelector
  --message "Gate pipeline build succeeded."
  --verified 2 --submit 458603,2

The Gerrit list has a post (Gerrit 2.14: Cannot post votes or downgrade votes on closed changes) that exactly describes it. You are not supposed to change votes after a change is closed. And it is said:

Labels can be configured to allow for post-submit votes using the allowPostSubmit setting. However, as explained there, you can't cast a vote score lower than the submit-time score, by design.

Given we vote verified 2 and it is the highest score, that should be fine.

''It used to be possible to bypass the limitation with --force-message but that has been removed a few Gerrit versions ago''

Task

Find out in Gerrit how to configure allowPostSubmit for the verified label.

Event Timeline

hashar created this task.Sep 7 2018, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2018, 9:02 PM
hashar updated the task description. (Show Details)Sep 7 2018, 9:03 PM
hashar added a comment.Sep 7 2018, 9:21 PM

I have tried on https://gerrit.wikimedia.org/r/#/c/test/gerrit-ping/+/458903/ a merged change with Verified+2

$ gerrit review  --verified 2 458903,1
$

Trying to submit:

$ gerrit review --submit 458903,1
error: fatal: change is merged

fatal: one or more reviews failed; review output above

:-\

hashar updated the task description. (Show Details)Sep 7 2018, 9:23 PM

Change 458914 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

hashar added a comment.Sep 7 2018, 9:55 PM

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

greg added a subscriber: greg.Sep 10 2018, 3:45 PM

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

We probably want to decide T94322 before putting too much more effort in forking Zuul in this way.

Change 470845 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

Change 458914 merged by Hashar:
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

Change 470845 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

greg assigned this task to hashar.Nov 21 2018, 12:12 AM
hashar moved this task from Backlog to Bugs on the Zuul board.Nov 26 2018, 10:54 PM
Paladox moved this task from Bugs & stuff to Local hacks on the Gerrit board.Dec 14 2018, 9:15 PM
hashar renamed this task from Zuul cancel all changes when a change is manually merged to Zuul cancels all changes when a change is manually merged.Dec 18 2018, 4:10 PM
hashar added a comment.Apr 4 2019, 8:44 PM

I have missed upgrading Zuul but should do it next monday (T208426).

I have upgraded Zuul to 2.5.1-wmf.6 (T208426). This is now pending verification which I guess can be done by looking at the Zuul error.log.

The patch has some issues though:

2019-04-11 00:55:50,226 DEBUG zuul.reporter.gerrit.Reporter: Report change <Change 0x7f2e586e6390 502894,5>, params {'verified': -1}, message: Gate pipeline build failed.

- quibble-vendor-mysql-hhvm-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/44406/console : FAILURE in 15m 47s
- quibble-vendor-mysql-php70-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/19662/console : FAILURE in 6m 45s
- quibble-vendor-mysql-php71-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php71-docker/8970/console : FAILURE in 5m 45s
- quibble-vendor-mysql-php72-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/8762/console : FAILURE in 6m 25s
- mwgate-npm-node-6-docker https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/97254/console : SUCCESS in 1m 16s
- mwext-php70-phan-seccheck-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker/49211/console : SUCCESS in 4m 05s
- mwext-php70-phan-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/25981/console : SUCCESS in 3m 12s
2019-04-11 00:55:50,299 ERROR zuul.DependentPipelineManager: Exception while reporting:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1768, in _reportItem
    ret = self.sendReport(actions, self.pipeline.source, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1293, in sendReport
    ret = reporter.report(source, self.pipeline, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/reporter/gerrit.py", line 39, in report
    message, self.reporter_config)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 287, in review
    if 'submit' not in action and 'change is merged' in err:
UnboundLocalError: local variable 'err' referenced before assignment

My code logic was wrong. The intent is to ignore error when trying to submit a patch that is already merged by returning early. But my patch had 'submit' not in action which does the opposite.

Changing to:

if 'submit' in action and err and 'change is merged' in err:

Change 502984 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

Change 502984 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

greg triaged this task as Normal priority.Apr 16 2019, 12:13 AM
hashar closed this task as Resolved.Apr 17 2019, 12:58 PM

Should be good now!

hashar reopened this task as Open.Sep 13 2019, 8:10 PM

The patch I wrote is wrong. In zuul/connection/gerrit.py:

def _ssh(...)
    ...
    ret = stdout.channel.recv_exit_status()
    err = stderr.read()
    if ret:
         # Some bad exit status
         raise Exception()
    return (out, err)

My patch catch the exception and look at the returned err:

try:
    out, err = self._ssh(cmd)
except Exception:
    if 'submit' in action and err and 'change is merged' in err:
         return

But since _ssh raises, nothing is return, err is never set and the code throws:

  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 287, in review
    if 'submit' in action and err and 'change is merged' in err:
UnboundLocalError: local variable 'err' referenced before assignment

So I guess we should enhance the exception thrown by _ssh to attach the command stderr and attach it to the exception, or potentially just make it the message.

raise Exception(err)

Change 536689 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix 0018-wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/536689

Change 536689 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix 0018-wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/536689

Would then need to rebuild the Debian package get it on apt.wikimedia.org and deploy.

Mentioned in SAL (#wikimedia-operations) [2019-09-20T08:12:01Z] <hashar> contint2001: upgrade zuul to 2.5.1-wmf10 # T203846

Mentioned in SAL (#wikimedia-operations) [2019-09-20T08:20:10Z] <hashar> contint1001: upgrade zuul to 2.5.1-wmf10 # T203846

So that looks like:

2019-09-24 02:23:33,376 ERROR zuul.DependentPipelineManager:
Reporting item <QueueItem 0x7f21185d0890 for <Change 0x7f2119459950 538725,1> in gate-and-submit> received: ['submit ignored on change already merged'] 

2019-09-24 02:23:33,382 INFO zuul.DependentPipelineManager:
Reported change <Change 0x7f2119459950 538725,1> status: all-succeeded: True, merged: False

2019-09-24 02:23:33,382 DEBUG zuul.DependentPipelineManager:
Reported change <Change 0x7f2119459950 538725,1> failed tests or failed to merge

2019-09-24 02:23:33,382 DEBUG zuul.DependentPipelineManager:
<ChangeQueue gate-and-submit: xxxbunch of reposxxx> window size decreased to 2

So instead of returning a message ('submit ignored on change already merged') we should just log and return nothing :-\