See, eg, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/examples/+/799464, which is literally just an empty patch
Merge Failed.
This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | hashar | T309376 Zuul jenkins-bot user holding open SSH sessions | |||
| Resolved | SLyngshede-WMF | T309371 Gerrit: all patches are being reported as merge conflicts |
Looks like T308943: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos again..? Raising to UBN 😣
On contint1001 I see the following in /var/log/zuul/merger-debug.log:
2022-05-27 04:36:23,233 DEBUG zuul.Merger: No base commit found for (u'machinelearning/liftwing/inference-services', u'main')
2022-05-27 04:36:23,233 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/machinelearning/liftwing/inference-services
2022-05-27 04:36:23,234 DEBUG zuul.Repo: Updating repository /srv/zuul/git/machinelearning/liftwing/inference-services
2022-05-27 04:36:23,391 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f3ad49c2050>
Traceback (most recent call last):
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 326, in _mergeItem
repo.reset()
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 101, in reset
self.update()
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 207, in update
origin.fetch(tags=True, force=True)
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 789, in fetch
res = self._get_fetch_info_from_stderr(proc, progress)
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/remote.py", line 675, in _get_fetch_info_from_stderr
proc.wait(stderr=stderr_text)
File "/srv/deployment/zuul/venv/local/lib/python2.7/site-packages/git/cmd.py", line 415, in wait
raise GitCommandError(self.args, status, errstr)
GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git fetch --force --tags -v origin
stderr: 'fatal: Could not read from remote repository.The same error repeats itself, the first occurrence that I see seems to be at 2022-05-27 03:22:56.
Indeed looks like the same issue as last time. The Gerrit logstash dashboard shows this:
Max connection count for user jenkins-bot exceeded, rejecting new connection. currentSessionCount = 4, maxSessionCount = 4
and Gerrit itself itself shows 3 open connections:
taavi@runko:~ $ ssh -p 29418 gerrit.wikimedia.org gerrit show-connections | grep jenkins 716fc32f jenkins-bot contint2001.wikimedi 944d965d jenkins-bot contint2001.wikimedi f45092f9 jenkins-bot contint2001.wikimedi
I've restarted Zuul on contint2001, and that seems to have helped a bit.
The Zuul service on contint1001 is currently masked/dead, and I'm a unsure if it is safe to just unmask it.
Yep, same error logs in logstash again.
Max connection count for user jenkins-bot exceeded, rejecting new connection. currentSessionCount = 4, maxSessionCount = 4
And same three connections from zuul in contint2001 as before too:
root@contint2001:~# netstat -tnp | grep 29418
tcp6 0 0 2620:0:860:1:208::39456 2620:0:861:2:208::29418 ESTABLISHED 15751/python2
tcp6 0 0 2620:0:860:1:208::57134 2620:0:861:2:208::29418 ESTABLISHED 15751/python2
tcp6 0 0 2620:0:860:1:208::57136 2620:0:861:2:208::29418 ESTABLISHED 15751/python2
root@contint2001:~# systemctl status 15751
● zuul.service - Zuul Server
Loaded: loaded (/lib/systemd/system/zuul.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-05-27 07:21:36 UTC; 1 months 15 days ago
Process: 30561 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 15751 (zuul-server)
Tasks: 29 (limit: 4915)
Memory: 1.8G
CGroup: /system.slice/zuul.service
├─15751 /srv/deployment/zuul/venv/bin/python2 /srv/deployment/zuul/venv/bin/zuul-server -d -c /etc/zuul/zuul-server.conf
└─15754 /srv/deployment/zuul/venv/bin/python2 /srv/deployment/zuul/venv/bin/zuul-server -d -c /etc/zuul/zuul-server.conf
Jul 12 08:06:41 contint2001 zuul-server[15751]: 2022-07-12 08:06:41,115 WARNING zuul.GerritEventConnector: Received unrecognized event type 'wip-state-changed' from Gerrit. Can not get account information.
Jul 12 08:06:51 contint2001 zuul-server[15751]: 2022-07-12 08:06:51,970 WARNING zuul.Scheduler: Build set <BuildSet item: <QueueItem 0x7f6f646b4c90 for <Change 0x7f6f8a378e10 727429,8> in coverage> #builds: 0 merge state: PENDING> is not current
Jul 12 08:07:23 contint2001 zuul-server[15751]: 2022-07-12 08:07:23,853 WARNING zuul.Scheduler: Build set <BuildSet item: <QueueItem 0x7f6f6464d490 for <Change 0x7f6f8a378e10 727429,8> in test> #builds: 0 merge state: PENDING> is not current
Jul 12 08:07:29 contint2001 zuul-server[15751]: 2022-07-12 08:07:29,799 WARNING zuul.Scheduler: Build set <BuildSet item: <QueueItem 0x7f6f646b4c10 for <Change 0x7f6f8a378e10 727429,8> in patch-performance> #builds: 0 merge state: PENDING> is not current
Jul 12 08:08:04 contint2001 zuul-server[15751]: 2022-07-12 08:08:04,348 WARNING zuul.Scheduler: Build set <BuildSet item: <QueueItem 0x7f6f646b4f50 for <Change 0x7f6f8a378e10 727429,8> in codehealth> #builds: 0 merge state: PENDING> is not current
Jul 12 08:15:52 contint2001 zuul-server[15751]: 2022-07-12 08:15:52,151 WARNING zuul.GerritEventConnector: Received unrecognized event type 'topic-changed' from Gerrit. Can not get account information.
Jul 12 08:23:56 contint2001 zuul-server[15751]: 2022-07-12 08:23:56,864 WARNING zuul.GerritEventConnector: Received unrecognized event type 'vote-deleted' from Gerrit. Can not get account information.
Jul 12 08:23:59 contint2001 zuul-server[15751]: 2022-07-12 08:23:59,540 WARNING zuul.GerritEventConnector: Received unrecognized event type 'vote-deleted' from Gerrit. Can not get account information.
Jul 12 08:24:25 contint2001 zuul-server[15751]: 2022-07-12 08:24:25,211 WARNING zuul.Scheduler: Build set <BuildSet item: <QueueItem 0x7f70002e5610 for <Change 0x7f70001fc290 812824,8> in gate-and-submit> #builds: 0 merge state: PENDING> is not current
Jul 12 08:33:56 contint2001 zuul-server[15751]: 2022-07-12 08:33:56,369 WARNING zuul.GerritEventConnector: Received unrecognized event type 'topic-changed' from Gerrit. Can not get account information.Mentioned in SAL (#wikimedia-operations) [2022-07-12T08:58:06Z] <hashar> Restarted Gerrit T309371
I have restarted Gerrit but that did not flush the Zuul server ssh connections to Gerrit.
I took a stack dump (kill -SIGUSR2 <pidof zuul>)
And the three connections:
lsof -p 15751|grep gerrit zuul-serv 15751 zuul 8u IPv6 295859537 0t0 TCP contint2001.wikimedia.org:57134->gerrit.wikimedia.org:29418 (ESTABLISHED) zuul-serv 15751 zuul 17u IPv6 173274822 0t0 TCP contint2001.wikimedia.org:39456->gerrit.wikimedia.org:29418 (ESTABLISHED) zuul-serv 15751 zuul 24u IPv6 295859542 0t0 TCP contint2001.wikimedia.org:57136->gerrit.wikimedia.org:29418 (ESTABLISHED)
Mentioned in SAL (#wikimedia-operations) [2022-07-12T09:12:29Z] <hashar> Restarted Zuul T309371
I have send SIGUSR1 to Zuul to let it finishes processing the ongoing job then restarted it systemctl restart zuul. We are back to two ssh connections:
$ ssh -p 29418 hashar@gerrit.wikimedia.org gerrit show-connections -w|grep jenkins-bot 9c367a9f jenkins-bot contint2001.wikimedia.org 9c619ab8 jenkins-bot contint2001.wikimedia.org
The first occurrence was I think at 6:39:41 UTC