Page MenuHomePhabricator

Zuul-merger: This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Closed, ResolvedPublic

Description

Something strange happening with zuul-merger on contint2001.

Superficially similar to T283665: Unable to fix "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset."

Was seeing

2023-04-10 18:41:48,390 DEBUG zuul.Merger: Unable to merge {u'oldrev': None, u'newrev': None, u'refspec': u'refs/changes/78/779578/6', u'merge_mode': 2, u'connection_name': u'gerrit', u'number': u'779578', u'proj
ect': u'labs/tools/graphql', u'url': u'ssh://jenkins-bot@gerrit.wikimedia.org:29418/labs/tools/graphql', u'branch': u'master', u'patchset': 6, u'ref': u'Z6c755d76c9dd4aa683cb2f3973781553'}
Traceback (most recent call last):                                                                                                                                                                                  
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 277, in _mergeChange
    commit = repo.merge(item['refspec'], 'resolve')                                                       
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 163, in merge
    self.fetch(ref)                                                                                       
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 176, in fetch
    origin.fetch(ref)                                                                                                                                                                                               
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 789, in fetch                                                                                                              
    res = self._get_fetch_info_from_stderr(proc, progress)        
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 675, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)                                                                         
  File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)                                                      
GitCommandError: Cmd('git') failed due to: exit code(128)              
  cmdline: git fetch -v origin refs/changes/78/779578/6                                                                                                                                                             
  stderr: 'fatal: Could not read from remote repository

But a manual git fetch worked as the zuul user sudo -u zuul git fetch -v

Restarted zuul-merger and it's still not working, but also now NOT raising an error in /var/log/zuul/merger-debug.log

Event Timeline

This does seem like a recurrence of T283665, based on logs?

But then again, I'm seeing logs for operations/puppet and DiscussionTools, so affecting more than one repo.

Went to try and check ownership in ops/puppet:

brennen@contint2001:/srv/zuul/git/operations/puppet$ git branch -a
fatal: detected dubious ownership in repository at '/srv/zuul/git/operations/puppet'
To add an exception for this directory, call:

        git config --global --add safe.directory /srv/zuul/git/operations/puppet

Something something permissions...

(Ignore that last bit, I was just forgetting a -u zuul.)

I think this is another instance of T309376: Zuul jenkins-bot user holding open SSH sessions

Restarted zuul Hoping that fixes it. Let's monitor.

thcipriani claimed this task.

After monitoring for a bit, seems like all these are working again. Declaring victory—please reopen if you spot new ones.