Page MenuHomePhabricator

Jenkins failing on all patches, "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository"
Closed, ResolvedPublic

Description

Jenkins is failing on all patches with the following message:

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

This is happening on all repositories, affected patches are already up-to-date with the main branch (usually master). Zuul is empty.

Event Timeline

Majavah triaged this task as Unbreak Now! priority.Apr 5 2021, 11:25 AM
Majavah created this task.

this is the error i see in gerrit

[2021-04-05T11:53:17.976+0000] [HTTP POST /r/changes/operations%2Fpuppet~676892/abandon (jiji from 154.57.12.229)] WARN  org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header: "Set-Cookie: WMF-Last-Access=05-Apr-2021;Path=/;HttpOnly;secure;Expires=Fri, 07 May 2021 00:00:00 GMT". Invalid 'expires' attribute: Fri, 07 May 2021 00:00:00 GMT
[2021-04-05T11:53:17.977+0000] [HTTP POST /r/changes/operations%2Fpuppet~676892/abandon (jiji from 154.57.12.229)] ERROR com.googlesource.gerrit.plugins.its.base.workflow.ActionExecutor : Error while executing action add-soy-comment PatchSetAbandoned [CONTEXT PLUGIN="gerrit" PLUGIN="its-phabricator" project="operations/puppet" ]
java.io.IOException: Could not add comment for task 278265
        at com.googlesource.gerrit.plugins.its.phabricator.PhabricatorItsFacade.addComment(PhabricatorItsFacade.java:53)
        at com.googlesource.gerrit.plugins.its.base.workflow.AddSoyComment.execute(AddSoyComment.java:85)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionExecutor.execute(ActionExecutor.java:88)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionExecutor.executeOnIssue(ActionExecutor.java:99)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionExecutor.executeOnIssue(ActionExecutor.java:108)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionController.handleIssuesEvent(ActionController.java:76)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionController.handleEvent(ActionController.java:68)
        at com.googlesource.gerrit.plugins.its.base.workflow.ActionController.onEvent(ActionController.java:60)
        at com.google.gerrit.server.events.EventBroker.lambda$fireEventForUnrestrictedListeners$0(EventBroker.java:111)
        at com.google.gerrit.server.plugincontext.PluginContext.runLogExceptions(PluginContext.java:206)
        at com.google.gerrit.server.plugincontext.PluginSetContext.lambda$runEach$1(PluginSetContext.java:148)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at com.google.gerrit.server.plugincontext.PluginSetContext.runEach(PluginSetContext.java:148)
        at com.google.gerrit.server.events.EventBroker.fireEventForUnrestrictedListeners(EventBroker.java:111)
        at com.google.gerrit.server.events.EventBroker.fireEvent(EventBroker.java:122)
        at com.google.gerrit.server.events.EventBroker.postEvent(EventBroker.java:91)
        at com.google.gerrit.server.events.StreamEventsApiListener.lambda$onChangeAbandoned$16(StreamEventsApiListener.java:448)
        at com.google.gerrit.server.plugincontext.PluginContext.runLogExceptions(PluginContext.java:206)
        at com.google.gerrit.server.plugincontext.PluginItemContext.run(PluginItemContext.java:131)
        at com.google.gerrit.server.events.StreamEventsApiListener.onChangeAbandoned(StreamEventsApiListener.java:448)
        at com.google.gerrit.server.extensions.events.ChangeAbandoned.lambda$fire$0(ChangeAbandoned.java:70)
        at com.google.gerrit.server.plugincontext.PluginContext.runLogExceptions(PluginContext.java:206)
        at com.google.gerrit.server.plugincontext.PluginSetContext.lambda$runEach$1(PluginSetContext.java:148)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at com.google.gerrit.server.plugincontext.PluginSetContext.runEach(PluginSetContext.java:148)
        at com.google.gerrit.server.extensions.events.ChangeAbandoned.fire(ChangeAbandoned.java:70)
        at com.google.gerrit.server.change.AbandonOp.postUpdate(AbandonOp.java:123)
        at com.google.gerrit.server.update.BatchUpdate.executePostOps(BatchUpdate.java:645)
        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:168)
        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:386)
        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:390)
        at com.google.gerrit.server.restapi.change.Abandon.abandon(Abandon.java:126)
        at com.google.gerrit.server.restapi.change.Abandon.apply(Abandon.java:81)
        at com.google.gerrit.server.restapi.change.Abandon.apply(Abandon.java:45)
        at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestModifyViewWithRetry$7(RestApiServlet.java:817)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:561)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:504)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:172)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:892)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestModifyViewWithRetry(RestApiServlet.java:812)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:517)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:386)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.ericsson.gerrit.plugins.goimport.GoImportFilter.doFilter(GoImportFilter.java:147)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.googlesource.gerrit.plugins.its.phabricator.conduit.ConduitErrorException: Method 'maniphest.edit' gave: ERR-CONDUIT-CORE, Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%B` conversion to escape binary strings data.
        at com.googlesource.gerrit.plugins.its.phabricator.conduit.ConduitConnection.call(ConduitConnection.java:123)
        at com.googlesource.gerrit.plugins.its.phabricator.conduit.Conduit.maniphestEdit(Conduit.java:123)
        at com.googlesource.gerrit.plugins.its.phabricator.PhabricatorItsFacade.addComment(PhabricatorItsFacade.java:51)
        ... 107 more

and on cont1001

2021-04-05 11:53:21,199 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f93f1a32ed0>
Traceback (most recent call last):
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 326, in _mergeItem
    repo.reset()
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 101, in reset
    self.update()
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 207, in update
    origin.fetch(tags=True, force=True)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 789, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 675, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)
GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch --force --tags -v origin
  stderr: 'fatal: Could not read from remote repository.

Please make sure you have the correct access rights

seems cont1001 is asking for confirmation for the gerrit.wikimedia.org fingerprint i would expect this to connect to the real hosts gerrit1001/2001. It seems that a noop change to the ssh module, which casued sshd to restarte surfaced this issues

zuul@contint1001:/srv/zuul/git/maps/kartotherian/deploy$ git fetch --force --tags -v origin
Warning: the RSA host key for 'gerrit.wikimedia.org' differs from the key for the IP address '2620:0:861:2:208:80:154:137'
Offending key for IP in /etc/ssh/ssh_known_hosts:736
Matching host key in /var/lib/zuul/.ssh/known_hosts:4
Are you sure you want to continue connecting (yes/no)?

Seems this is related to T171165 and T240266. The main issue is that we have two ssh daemons on the same host. one for gerrit which listens on port 29418 and the main ssh daemon. This means we have to ssh fingerprints for the gerrit hosts. for zuul we have hard-coded the fingerprint for the gerrit vip 29418. however we can still get conflicts, due to ssh checking the ssh key for gerrit.wikimedia.org on port 29418 with the ssh key for the ip address gerrit.wikimedia.org resolves to which matches the ssh key for the main ssh damon running on port 22. One will see an error message simlar to above when this happens. for now i have added the following config to ssh however a more permanent fix is required

Host gerrit.wikimedia.org
   CheckHostIP no
NOTE: should also check the following modules/gerrit/files/homedir/.ssh/known_hosts

looking into this now, it looks like the error in gerrit may be a red-herring (something to do with maniphest.edit which is for tasks? will investigate after we get zuul running)

jbond lowered the priority of this task from Unbreak Now! to Medium.Apr 5 2021, 1:32 PM

changed priority as there is a work around in place now

I noticed the ssh change also changed the known_hosts file

-gerrit1001.wikimedia.org,gerrit1001
,208.80.154.136,2620:0:861:2:208:80:154:136 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqijP1wFTzWToCRaz48A++WaG8Q9O5zoNF
XpDdrCUVt5U2CE2w70gg1IcvjqiFqY/KACFpjGTQ3jDwc8fRWYgE=
+gerrit1001.wikimedia.org,gerrit1001
,208.80.154.136,2620:0:861:2:208:80:154:137 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqijP1wFTzWToCRaz48A++WaG8Q9O5zoNF
XpDdrCUVt5U2CE2w70gg1IcvjqiFqY/KACFpjGTQ3jDwc8fRWYgE=

which was introduced in the 675124, and caused because of the following

$ sudo facter -p ipaddress6 networking.ip6                                                                               
ipaddress6 => 2620:0:861:2:208:80:154:136
networking.ip6 => 2620:0:861:2:208:80:154:137

Change 676933 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] ssh::server us legacy facts for ip addresses

https://gerrit.wikimedia.org/r/676933

FTR, this is still broken. Not sure if anybody noticed, given that the priority was lowered to normal. See r676890 for an example (5 mins ago).

changed priority as there is a work around in place now

Where can I find the work around?

Cannot commit anything.
See https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/676923

Xqt raised the priority of this task from Medium to Unbreak Now!.Apr 5 2021, 2:27 PM
thcipriani lowered the priority of this task from Unbreak Now! to Medium.Apr 5 2021, 2:53 PM

FTR, this is still broken. Not sure if anybody noticed, given that the priority was lowered to normal. See r676890 for an example (5 mins ago).

This fix was only applied to contint1001, should be on contint2001 now: I re+2'd that patch and it works now.

changed priority as there is a work around in place now

Where can I find the work around?

Cannot commit anything.
See https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/676923

Should be working again now (fix was only applied on 1/2 the hosts) please re-run l10nbot or +2 and it should work.

Should be working again now (fix was only applied on 1/2 the hosts) please re-run l10nbot or +2 and it should work.

Works again. Great, thank you!

jbond claimed this task.

I have now merged the change to export the correct ipv6 address for ssh_know_hosts deployed the change to gerrit1001, contint1001 & contint2001. i have also removed the work temporary workaround and tested that git still works for the zuul user

$ git fetch --force --tags -v origin
From ssh://gerrit.wikimedia.org:29418/maps/kartotherian/deploy
 = [up to date]          imposm     -> origin/imposm
 = [up to date]          master     -> origin/master
 = [up to date]          new_deploy -> origin/new_deploy
 = [up to date]          node10     -> origin/node10
 = [up to date]          stretch    -> origin/stretch
 = [up to date]          sync-repo  -> origin/sync-repo
$ rm ~/.ssh/config 
$ git fetch --force --tags -v origin
RSA host key for IP address '2620:0:861:2:208:80:154:137' not in list of known hosts.
From ssh://gerrit.wikimedia.org:29418/maps/kartotherian/deploy
 = [up to date]          imposm     -> origin/imposm
 = [up to date]          master     -> origin/master
 = [up to date]          new_deploy -> origin/new_deploy
 = [up to date]          node10     -> origin/node10
 = [up to date]          stretch    -> origin/stretch
 = [up to date]          sync-repo  -> origin/sync-repo

Will resolve the ticket but please re-open if you still see issues