Page MenuHomePhabricator
Create Task
Maniphest T203846

Zuul cancels all changes when a change is manually merged
Closed, ResolvedPublic
Actions

Description

Given gate-and-submit has changes A <- B <- C each having jobs running

A user manually merges A, The jobs are still running in Zuul

When jobs are completed, Zuul submits the change with --verified +2

Gerrit reject the review since it tries to vote on a closed change

Zuul considers A failed to merge. It thus cancel jobs of B and C and rebuild the queue.

Result: waste of time.

Example with https://gerrit.wikimedia.org/r/c/458603 . /var/log/zuul/error.log has:

Exception: Gerrit error executing gerrit review
  --project mediawiki/extensions/UniversalLanguageSelector
  --message "Gate pipeline build succeeded."
  --verified 2 --submit 458603,2

The Gerrit list has a post (Gerrit 2.14: Cannot post votes or downgrade votes on closed changes) that exactly describes it. You are not supposed to change votes after a change is closed. And it is said:

Labels can be configured to allow for post-submit votes using the allowPostSubmit setting. However, as explained there, you can't cast a vote score lower than the submit-time score, by design.

Given we vote verified 2 and it is the highest score, that should be fine.

''It used to be possible to bypass the limitation with --force-message but that has been removed a few Gerrit versions ago''

Task

Find out in Gerrit how to configure allowPostSubmit for the verified label.

Details

SubjectRepoBranchLines +/-
Fix ignore-submit-error-on-merged-change patchintegration/zuuldebian/jessie-wikimedia+18 -13
Fix 0018-wmf-ignore-submit-error-on-merged-changeintegration/zuuldebian/jessie-wikimedia+29 -12
Fix flaw in wmf-ignore-submit-error-on-merged-changeintegration/zuuldebian/jessie-wikimedia+13 -4
2.5.1-wmf5: maintenance patchesintegration/zuuldebian/jessie-wikimedia+83 -0
wmf: ignore submit error on merged changeintegration/zuulpatch-queue/debian/jessie-wikimedia+8 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Sep 7 2018, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2018, 9:02 PM
hashar updated the task description. (Show Details)Sep 7 2018, 9:03 PM
hashar added a comment.Sep 7 2018, 9:21 PM

I have tried on https://gerrit.wikimedia.org/r/#/c/test/gerrit-ping/+/458903/ a merged change with Verified+2

$ gerrit review  --verified 2 458903,1
$

Trying to submit:

$ gerrit review --submit 458903,1
error: fatal: change is merged

fatal: one or more reviews failed; review output above

:-\

hashar updated the task description. (Show Details)Sep 7 2018, 9:23 PM
gerritbot subscribed.Sep 7 2018, 9:51 PM

Change 458914 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

gerritbot added a project: Patch-For-Review.Sep 7 2018, 9:51 PM
hashar added a comment.Sep 7 2018, 9:55 PM

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

greg subscribed.Sep 10 2018, 3:45 PM
In T203846#4568265, @hashar wrote:

I don't think that can be fixed in Gerrit short of reintroducing force message. It is easier to monkey patch Zuul

We probably want to decide T94322 before putting too much more effort in forking Zuul in this way.

gerritbot added a comment.Oct 31 2018, 2:37 PM

Change 470845 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

gerritbot added a comment.Oct 31 2018, 2:38 PM

Change 458914 merged by Hashar:
[integration/zuul@patch-queue/debian/jessie-wikimedia] wmf: ignore submit error on merged change

https://gerrit.wikimedia.org/r/458914

gerritbot added a comment.Oct 31 2018, 2:42 PM

Change 470845 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] 2.5.1-wmf5: maintenance patches

https://gerrit.wikimedia.org/r/470845

hashar mentioned this in T208426: Deploy Zuul 2.5.1-wmf6.Oct 31 2018, 2:42 PM
greg assigned this task to hashar.Nov 21 2018, 12:12 AM
Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team. · View Herald TranscriptNov 21 2018, 12:12 AM
greg moved this task from Backlog to Needs review (internally) on the Release-Engineering-Team (Kanban) board.Nov 21 2018, 12:12 AM
hashar moved this task from Backlog to Bugs on the Zuul board.Nov 26 2018, 10:54 PM
Paladox moved this task from Bugs & stuff to WMF customizations on the Gerrit board.Dec 14 2018, 9:15 PM
hashar renamed this task from Zuul cancel all changes when a change is manually merged to Zuul cancels all changes when a change is manually merged.Dec 18 2018, 4:10 PM
hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.Jan 1 2019, 5:31 PM
hashar added a comment.Apr 4 2019, 8:44 PM

I have missed upgrading Zuul but should do it next monday (T208426).

hashar added a comment.Apr 8 2019, 12:29 PM

I have upgraded Zuul to 2.5.1-wmf.6 (T208426). This is now pending verification which I guess can be done by looking at the Zuul error.log.

hashar closed subtask T208426: Deploy Zuul 2.5.1-wmf6 as Resolved.Apr 8 2019, 12:30 PM
hashar added a comment.Apr 11 2019, 12:18 PM

The patch has some issues though:

2019-04-11 00:55:50,226 DEBUG zuul.reporter.gerrit.Reporter: Report change <Change 0x7f2e586e6390 502894,5>, params {'verified': -1}, message: Gate pipeline build failed.

- quibble-vendor-mysql-hhvm-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/44406/console : FAILURE in 15m 47s
- quibble-vendor-mysql-php70-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/19662/console : FAILURE in 6m 45s
- quibble-vendor-mysql-php71-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php71-docker/8970/console : FAILURE in 5m 45s
- quibble-vendor-mysql-php72-docker https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/8762/console : FAILURE in 6m 25s
- mwgate-npm-node-6-docker https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/97254/console : SUCCESS in 1m 16s
- mwext-php70-phan-seccheck-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker/49211/console : SUCCESS in 4m 05s
- mwext-php70-phan-docker https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/25981/console : SUCCESS in 3m 12s
2019-04-11 00:55:50,299 ERROR zuul.DependentPipelineManager: Exception while reporting:
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1768, in _reportItem
    ret = self.sendReport(actions, self.pipeline.source, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 1293, in sendReport
    ret = reporter.report(source, self.pipeline, item)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/reporter/gerrit.py", line 39, in report
    message, self.reporter_config)
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 287, in review
    if 'submit' not in action and 'change is merged' in err:
UnboundLocalError: local variable 'err' referenced before assignment
hashar added a comment.Apr 11 2019, 12:27 PM

My code logic was wrong. The intent is to ignore error when trying to submit a patch that is already merged by returning early. But my patch had 'submit' not in action which does the opposite.

Changing to:

if 'submit' in action and err and 'change is merged' in err:
gerritbot added a comment.Apr 11 2019, 12:30 PM

Change 502984 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

gerritbot added a comment.Apr 11 2019, 12:56 PM

Change 502984 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix flaw in wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/502984

greg triaged this task as Medium priority.Apr 16 2019, 12:13 AM
hashar mentioned this in T221227: Deploy Zuul 2.5.1-wmf7.Apr 17 2019, 12:17 PM
hashar closed this task as Resolved.Apr 17 2019, 12:58 PM

Should be good now!

hashar reopened this task as Open.Sep 13 2019, 8:10 PM

The patch I wrote is wrong. In zuul/connection/gerrit.py:

def _ssh(...)
    ...
    ret = stdout.channel.recv_exit_status()
    err = stderr.read()
    if ret:
         # Some bad exit status
         raise Exception()
    return (out, err)

My patch catch the exception and look at the returned err:

try:
    out, err = self._ssh(cmd)
except Exception:
    if 'submit' in action and err and 'change is merged' in err:
         return

But since _ssh raises, nothing is return, err is never set and the code throws:

  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 287, in review
    if 'submit' in action and err and 'change is merged' in err:
UnboundLocalError: local variable 'err' referenced before assignment

So I guess we should enhance the exception thrown by _ssh to attach the command stderr and attach it to the exception, or potentially just make it the message.

raise Exception(err)
Maintenance_bot removed a project: Patch-For-Review.Sep 13 2019, 8:10 PM
gerritbot added a comment.Sep 13 2019, 9:12 PM

Change 536689 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix 0018-wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/536689

gerritbot added a project: Patch-For-Review.Sep 13 2019, 9:12 PM
gerritbot added a comment.Sep 13 2019, 9:19 PM

Change 536689 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix 0018-wmf-ignore-submit-error-on-merged-change

https://gerrit.wikimedia.org/r/536689

hashar added a comment.Sep 13 2019, 9:19 PM

Would then need to rebuild the Debian package get it on apt.wikimedia.org and deploy.

Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201909); removed Release-Engineering-Team (Kanban).Sep 13 2019, 9:23 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 13 2019, 10:10 PM
hashar mentioned this in T233025: Upload zuul_2.5.1-wmf10 to apt.wikimedia.org.Sep 16 2019, 3:09 PM
hashar added a subtask: T233025: Upload zuul_2.5.1-wmf10 to apt.wikimedia.org.Sep 18 2019, 8:49 AM
herron closed subtask T233025: Upload zuul_2.5.1-wmf10 to apt.wikimedia.org as Resolved.Sep 18 2019, 7:08 PM
Stashbot added a comment.Sep 20 2019, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-20T08:12:01Z] <hashar> contint2001: upgrade zuul to 2.5.1-wmf10 # T203846

Stashbot added a comment.Sep 20 2019, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-20T08:20:10Z] <hashar> contint1001: upgrade zuul to 2.5.1-wmf10 # T203846

hashar mentioned this in T233390: zuul-merger fails to fetch from Gerrit.Sep 20 2019, 8:30 AM
hashar added a comment.Sep 26 2019, 10:27 AM

So that looks like:

2019-09-24 02:23:33,376 ERROR zuul.DependentPipelineManager:
Reporting item <QueueItem 0x7f21185d0890 for <Change 0x7f2119459950 538725,1> in gate-and-submit> received: ['submit ignored on change already merged'] 

2019-09-24 02:23:33,382 INFO zuul.DependentPipelineManager:
Reported change <Change 0x7f2119459950 538725,1> status: all-succeeded: True, merged: False

2019-09-24 02:23:33,382 DEBUG zuul.DependentPipelineManager:
Reported change <Change 0x7f2119459950 538725,1> failed tests or failed to merge

2019-09-24 02:23:33,382 DEBUG zuul.DependentPipelineManager:
<ChangeQueue gate-and-submit: xxxbunch of reposxxx> window size decreased to 2

So instead of returning a message ('submit ignored on change already merged') we should just log and return nothing :-\

hashar edited projects, added Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO (201909).Oct 1 2019, 3:44 PM
Jdforrester-WMF moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201910) board.Oct 16 2019, 5:21 PM
Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201911); removed Release-Engineering-Team-TODO (201910).Nov 6 2019, 4:23 PM
Jdforrester-WMF moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201911) board.
Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201912); removed Release-Engineering-Team-TODO (201911).Dec 2 2019, 7:32 PM
Jdforrester-WMF moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201912) board.
gerritbot added a comment.Dec 9 2019, 8:54 PM

Change 556054 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/zuul@debian/jessie-wikimedia] Fix ignore-submit-error-on-merged-change patch

https://gerrit.wikimedia.org/r/556054

gerritbot added a project: Patch-For-Review.Dec 9 2019, 8:54 PM
hashar added a comment.Dec 9 2019, 8:56 PM

Eventually I forgot about this task. The fix I have done previously was not complete: the reporter should return nothing, not a message which is considered an error.

greg unsubscribed.Dec 9 2019, 11:26 PM
gerritbot added a comment.Dec 12 2019, 10:55 AM

Change 556054 merged by jenkins-bot:
[integration/zuul@debian/jessie-wikimedia] Fix ignore-submit-error-on-merged-change patch

https://gerrit.wikimedia.org/r/556054

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2019, 11:11 AM
Stashbot added a comment.Dec 12 2019, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-12T13:38:51Z] <hashar> contint1001 / contint2001 : upgraded Zuul to 2.5.1-wmf11 # T203846

hashar closed this task as Resolved.Dec 12 2019, 1:42 PM
hashar mentioned this in T240570: Upload zuul_2.5.1-wmf11 to apt.wikimedia.org.

I have deployed the fix again.

hashar closed subtask T240570: Upload zuul_2.5.1-wmf11 to apt.wikimedia.org as Resolved.Dec 13 2019, 10:45 AM
Jdforrester-WMF moved this task from Ready to Completed on the Release-Engineering-Team-TODO (201912) board.Dec 17 2019, 5:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits