Maniphest T420865

Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Reedy
	Mar 22 2026, 11:29 PM

Project Tags

Referenced Files

	F87756468: image.png
	Jun 10 2026, 7:07 AM

Subscribers

View All 25 Subscribers

Description

Git fetch/clones operations are being aborted with git/curl reporting:

GnuTLS recv error (-54): Error in the pull function

In T421827#11785472, @A_smart_kitten wrote:
17:10:06   stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/CentralAuth/': GnuTLS recv error (-54): Error in the pull function.'
To be clear, this is basically just speculation based on the limited information I can find; but I am wondering about whether this type of error might potentially be unrelated to the addition of Envoy to Gerrit's stack.
Searching Gerrit for the text GnuTLS recv error (-54) (to try and find instances where this CI error has been pasted into a Gerrit comment), it seems like an early singular occurrence of this issue may have been on 2026-02-18, with occurrences starting regularly from 2026-03-16 onwards. IIUC from T420909#11752446, Envoy was added to Gerrit's stack on 2026-03-26.

The Gerrit search for "GnuTLS recv error" shows a person asking CI to be rechecked on Feb 18th.

Some history about Gerrit

Feb 10th: switch behind the CDN (T411895)
Feb 16th: Gerrit moved from eqiad to codfw (which caused 502 http errors due to a timeout race condition made more frequents with the increased latency)
March 16th start of several recheck messages mentioning GnuTLS recv error
March 22nd this task get filed
March 26 Envoy is introduced (T420909 / Puppet #1259945)
April 30th/May 3rd). The issue has been very prominent during the Hackathon Probably because lot of patches were sent which made the issue appear more frequently or some system is overloaded.

Quibble stacktrace

23:12:20 Traceback (most recent call last):
23:12:20   File "/usr/local/bin/quibble", line 6, in <module>
23:12:20     sys.exit(main())
23:12:20   File "/usr/local/lib/python3.9/dist-packages/quibble/cmd.py", line 1017, in main
23:12:20     cmd.execute(
23:12:20   File "/usr/local/lib/python3.9/dist-packages/quibble/cmd.py", line 640, in execute
23:12:20     quibble.commands.execute_command(command)
23:12:20   File "/usr/local/lib/python3.9/dist-packages/quibble/commands.py", line 36, in execute_command
23:12:20     command.execute()
23:12:20   File "/usr/local/lib/python3.9/dist-packages/quibble/commands.py", line 304, in execute
23:12:20     quibble.zuul.clone(
23:12:20   File "/usr/local/lib/python3.9/dist-packages/quibble/zuul.py", line 96, in clone
23:12:20     zuul_cloner.prepareRepo('mediawiki/core', dests['mediawiki/core'])
23:12:20   File "/usr/local/lib/python3.9/dist-packages/zuul/lib/cloner.py", line 167, in prepareRepo
23:12:20     repo.reset()
23:12:20   File "/usr/local/lib/python3.9/dist-packages/zuul/merger/merger.py", line 99, in reset
23:12:20     self.update()
23:12:20   File "/usr/local/lib/python3.9/dist-packages/zuul/merger/merger.py", line 205, in update
23:12:20     origin.fetch(tags=True, force=True)
23:12:20   File "/usr/lib/python3/dist-packages/git/remote.py", line 831, in fetch
23:12:20     res = self._get_fetch_info_from_stderr(proc, progress)
23:12:20   File "/usr/lib/python3/dist-packages/git/remote.py", line 698, in _get_fetch_info_from_stderr
23:12:20     proc.wait(stderr=stderr_text)
23:12:20   File "/usr/lib/python3/dist-packages/git/cmd.py", line 455, in wait
23:12:20     raise GitCommandError(self.args, status, errstr)
23:12:20 git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
23:12:20   cmdline: git fetch --force --tags -v -- origin
23:12:20   stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/core/': GnuTLS recv error (-54): Error in the pull function.'

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
ci: add repositories to gitcache	Arnaudb	operations/puppet	production	+49 -0
Upgrade Quibble jobs to 1.18.2	Ahmon Dancy	integration/config	master	+33 -33
dockerfiles: update Quibble to 1.18.2	Ahmon Dancy	integration/config	master	+49 -1
zuul: retry on connection resets, not only GnuTLS errors	Arnaudb	integration/quibble	master	+12 -4
ci: update jenkins build monitor	Arnaudb	operations/puppet	production	+145 -42
ci: monitor a wider variety of network errors	Arnaudb	operations/puppet	production	+45 -9
jjb: Upgrade Quibble jobs to 1.18.1	Thcipriani	integration/config	master	+33 -33
zuul: retry policy on network errors	Arnaudb	integration/quibble	master	+49 -6
jjb: enable git curl trace dump for Quibble	Hashar	integration/config	master	+39 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T393239 ProbeDown
		Restricted Task
		Restricted Task
		Restricted Task
Resolved	Dzahn	T411895 gerrit behind CDN
Resolved	None	T411904 ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation)
Resolved	CDanis	T414719 Opt-in testing of Gerrit-via-CDN
Resolved	ABran-WMF	T414940 Handle httpd log surplus coming from Liberica
Resolved	Jelto	T416189 Add monitoring and alerting for gerrits tcp-proxy
Resolved	Dzahn	T408532 Deploy a TCP proxy across all DCs
Resolved	Dzahn	T408064 Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy)
Resolved	Jelto	T417156 Pushing to gerrit over http is blocked by generic rate limiting
Resolved	Vgutierrez	T417536 ATS causes git fetches from Gerrit to fail with 502 responses
Resolved	ABran-WMF	T418108 gerrit: move gerrit-replica behind CDN
Resolved	ABran-WMF	T417998 ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use
Resolved	ABran-WMF	T420189 Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow
Resolved	ABran-WMF	T420909 gerrit: Add Envoy in Gerrit's stack
Resolved	ABran-WMF	T418361 gerrit: move gerrit-spare behind CDN
		Restricted Task
		Restricted Task
Resolved	ABran-WMF	T420677 GerritHAProxyBackendUnavailable - Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh
		Restricted Task
		Restricted Task
		Restricted Task
		Restricted Task
Open	None	T392448 Upgrade to Gerrit 3.12
Open	None	T379714 Upgrade to Gerrit 3.11
Open	None	T392465 Switch Gerrit from Java 17 to Java 21
Resolved	ABran-WMF	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
Resolved	ABran-WMF	T387833 Gerrit switchover process
Open	None	T388507 Investigate missing commit from old changes
Resolved	ABran-WMF	T260666 Create a cookbook to automate gerrit's switchover
Resolved	ABran-WMF	T393050 ProbeDown
Resolved	MatthewVernon	T392186 Grant Gerrit admin to arnaudb
Resolved	LSobanski	T393034 Investigate out of date refs following gerrit switchover
Resolved	ABran-WMF	T395440 Gerrit read-only plugin test
Resolved	ABran-WMF	T406334 Gerrit switchover between secondary instances
Resolved	LSobanski	T406482 SystemdUnitFailed (jenkins.service on contint1002)
Resolved	hashar	T406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
Resolved	hashar	T411583 Gerrit backups are growing
Resolved	hashar	T430774 Compact Gerrit H2 databases to reclaim disk
Open	None	T430775 Validate Gerrit backup recoverability after the fileset exclusions
Open	None	T433926 Backup Gerrit with LVM snapshots and Bacula pre/post job hooks
Resolved	hashar	T406856 Reduce size of analytics/superset/deploy.git Gerrit repo
Resolved	ABran-WMF	T407200 Gerrit is down
Resolved	ABran-WMF	T412779 Gerrit failover process
Resolved	ABran-WMF	T417246 Reimage gerrit1003
Resolved	dancy	T417767 scap error on Gerrit first setup
Resolved	ABran-WMF	T417777 SystemdUnitFailed - apache2.service on gerrit1003:9100
Duplicate	None	T417968 ProbeDown
Duplicate	ABran-WMF	T417966 GerritHAProxyBackendUnavailable
Resolved	ABran-WMF	T417247 Reimage gerrit2002
Resolved	ABran-WMF	T418242 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
Resolved	ABran-WMF	T418264 Fix gerrit-restart cookbook
Resolved	ABran-WMF	T418266 GerritHAProxyServiceUnavailable
Resolved	ABran-WMF	T418240 GerritReplicationUnavailable
Resolved	ABran-WMF	T417279 Harmonize DNS on all gerrit instances
Resolved	ABran-WMF	T417565 GerritHAProxyBackendUnavailable
Resolved	ABran-WMF	T417566 GerritHAProxyServiceUnavailable
Resolved	Dzahn	T372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
Resolved	ABran-WMF	T338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
Resolved	Dzahn	T379213 PuppetFailure - gerrit2003
Open	None	T417642 Rename gerrit2 Gerrit user to gerrit in the administrator group
Resolved	Jelto	T333143 Move Gerrit data out of root partition
Resolved	Jelto	T424924 GerritHAProxyBackendUnavailable
Resolved	Marostegui	T388785 SSH on gerrit2003 is CRITICAL: CRITICAL
Resolved	Dzahn	T374899 PuppetFailure - gerrit2003
		Restricted Task
Resolved	Dzahn	T402847 ProbeDown - gerrit1003
Resolved	ABran-WMF	T403347 ProbeDown
Resolved	ABran-WMF	T403963 ProbeDown - http_gerrit_tls_ip4 gerrit2003
Resolved	ABran-WMF	T404070 ProbeDown - http_gerrit_tls_ip4 gerrit2003.
		Restricted Task
Duplicate	None	T431446 GerritHAProxyBackendUnavailable
Duplicate	None	T431447 GerritHAProxyServiceUnavailable
Duplicate	None	T431448 ProbeDown
		Restricted Task
		Restricted Task
Resolved	dancy	T421736 Daily mediawiki-core-phpmetrics job broken since March 26, 2026
Resolved	ABran-WMF	T421827 gerrit: Adapt timeouts to avoid 502 errors in CI jobs
Duplicate	ABran-WMF	T424990 Implement a retry policy for network errors in CI
Resolved	ABran-WMF	T420865 Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function
Resolved	ABran-WMF	T429749 Gerrit: network errors in CI
Resolved	hashar	T246763 Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos
Resolved	ABran-WMF	T420311 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
Open	None	T430901 Get traces/logs from ATS on Gerrit CI errors

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Adding in @ABran-WMF as this is new and so is T420909: gerrit: Add Envoy in Gerrit's stack which changed TLS termination for Gerrit.

The timing of the Envoy change doesn't match up with this ticket being filed. But it is something beyond Apache (I think) causing this error.

Explainer

The error is pretty generic. My rough understanding is: something hung up the call. Maybe a timeout along the way.

My understanding of the path back to Jenkins is: [gerrit] -> [apache] -> [envoy] -> [CDN] -> [Jenkins]
And I've been able to trace the path through: [gerrit] -> [apache]

And then I dead end.

Details

The error seen here: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/23473/console

is:

08:54:12 INFO:zuul.Cloner.mediawiki/extensions/WikiLove:Creating repo mediawiki/extensions/WikiLove from upstream https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLove
...
08:54:19 git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
08:54:19   cmdline: git fetch --force --tags -v -- origin
08:54:19   stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLove/': GnuTLS recv error (-54): Error in the pull function.'

And here's what I see in the logs for that:

root@gerrit2003:~# grep -E '2026-03-26T14:54:12' /var/log/apache2/gerrit.wikimedia.org.https.access.log | grep 'git/2' | cut -c7- | jq -r '"\(.timestamp): \(."url.path") \(."http.request.method") \(."http.response.status_code") \(."http.response.bytes")"' | grep Love
2026-03-26T14:54:12: /r/mediawiki/extensions/WikiLove/info/refs GET 200 113
2026-03-26T14:54:12: /r/mediawiki/extensions/WikiLove/git-upload-pack POST 200 3183
2026-03-26T14:54:12: /r/mediawiki/extensions/WikiLove/git-upload-pack POST 200 3091437

Which looks normal to me for a the server side of a git pull (GET info/refs, 1×POST git-upload-pack for ls-refs, 1×POST to git-upload-pack for fetch).

This, for me, means that we're hitting some timeout in the reminder of the path and giving up. And since this ticket pre-dates Envoy...I guess the CDN is hanging up? No idea how to verify that.

There were timeout issues over the week-end but I don't quite remember the details.

The series of issues that are surging now are related to the introduction of Envoy which was done this morning and its timeout with the Apache backend is superior (I believe the default: 5 minutes) to the one on Apache side (122 seconds). Thus Envoy reuse one of those connection and while the packet travel Apache on the other side terminates the connection because it has exceeded the Apache timeout.

I'll mark this as a dupe of T420909 and comment there.

hashar closed this task as a duplicate of T420909: gerrit: Add Envoy in Gerrit's stack.Mar 26 2026, 8:51 PM

hashar mentioned this in T420909: gerrit: Add Envoy in Gerrit's stack.Mar 26 2026, 9:11 PM

Change #1283846 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: enable git curl trace dump for Quibble

https://gerrit.wikimedia.org/r/1283846

gerritbot added a project: Patch-For-Review.May 6 2026, 5:09 PM

I have updated the two heavy jobs which sets GIT_TRACE_CURL (Gerrit #1283846. The environment variable causes git to write down a debug log of curl operations. The log is discarded unless the build is in FAILURE in which case the gzipped trace is attached to the build as quibble-git-trace-curl.log.gz.

The jobs are:
https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/
https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/

Build that failed can be found using over the last hour can be found with:

ssh contint.wikimedia.org \
  find /srv/jenkins/builds -maxdepth 3 -name log -mmin -60 -exec grep -l GnuTLS  {} +

Once we got a failure, we can look at the curl debug log.

I have rolled back both jobs. I got a 179 MB gzipped curl trace at https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/32927/ . Save on contint1002 in /home/hashar/quibble-git-trace-curl.log.gz

The git clone commands are run in parallel and thus the log is multiplexing those traces

10:11:44.870103 http.c:715              => Send header: GET /r/mediawiki/extensions/Poem/info/refs?service=git-upload-pack HTTP/2
...
10:11:45.077553 http.c:756              == Info: GnuTLS recv error (-54): Error in the pull function.
10:11:45.077572 http.c:756              == Info: Failed receiving HTTP2 data
10:11:45.077586 http.c:756              == Info: Failed sending HTTP2 data
10:11:45.077596 http.c:756              == Info: Connection #0 to host gerrit.wikimedia.org left intact

dancy subscribed.May 7 2026, 5:08 PM

A_smart_kitten mentioned this in T424990: Implement a retry policy for network errors in CI.May 8 2026, 12:05 PM

Novem_Linguae subscribed.May 8 2026, 12:10 PM

Jdlrobson-WMF subscribed.May 15 2026, 6:02 PM

hashar renamed this task from Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Error in the pull function.'" to Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function.May 18 2026, 9:36 AM

hashar updated the task description. (Show Details)

A_smart_kitten updated the task description. (Show Details)May 18 2026, 10:10 AM

Change #1283846 abandoned by Hashar:

[integration/config@master] jjb: enable git curl trace dump for Quibble

Reason:

Debug logs of each `git clone` commands end up being multiplexed in a single log file :/

https://gerrit.wikimedia.org/r/1283846

Maintenance_bot removed a project: Patch-For-Review.May 18 2026, 2:31 PM

ABran-WMF added a parent task: T424990: Implement a retry policy for network errors in CI.May 20 2026, 7:30 AM

ABran-WMF moved this task from Incoming to Work in Progress (Tracking tasks) on the Collaboration-Services board.May 20 2026, 7:34 AM

Change #1278483 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[integration/quibble@master] zuul: retry policy on network errors

https://gerrit.wikimedia.org/r/1278483

gerritbot added a project: Patch-For-Review.May 20 2026, 7:45 AM

Michael subscribed.May 20 2026, 10:33 AM

A_smart_kitten merged a task: T424990: Implement a retry policy for network errors in CI.May 20 2026, 9:46 PM

A_smart_kitten added subscribers: Peter, WMDE-Fisch, Zabe, Dreamy_Jazz.

Nemoralis subscribed.May 21 2026, 11:23 PM

siebrand subscribed.May 22 2026, 9:08 AM

Change #1278483 had a related patch set uploaded (by Thcipriani; author: Arnaudb):

[integration/quibble@master] zuul: retry policy on network errors

https://gerrit.wikimedia.org/r/1278483

Definitely still happening despite the above patch, e.g. https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/36425/console this morning.

Oh wait, that was the same patch appearing twice as "uploaded" for some reason, not getting merged. So, ignore that "despite the patch" comment.

• matmarex subscribed.Jun 2 2026, 12:32 PM

Change #1278483 merged by jenkins-bot:

[integration/quibble@master] zuul: retry policy on network errors

https://gerrit.wikimedia.org/r/1278483

The patch by Arnaud to add retries has now been merged thanks to Tyler.

But let me copy over his comments from Gerrit right here:

Tyler wrote:

"Noting that after merge, needs a subsequent quibble release and rebuild of images for CI jobs before it's active."

"This change will attempt a git retry only for CI jobs which use quibble (i.e., MediaWiki jobs); GnuTLS failures in other jobs will require similar workarounds. I believe there's a problem at the edge that would be worth investigating, still, after this change is deployed."

Change #1296700 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[integration/config@master] Upgrade Quibble jobs to 1.18.1

https://gerrit.wikimedia.org/r/1296700

Change #1296700 merged by jenkins-bot:

[integration/config@master] jjb: Upgrade Quibble jobs to 1.18.1

https://gerrit.wikimedia.org/r/1296700

In T420865#11979145, @Dzahn wrote:

Tyler wrote:

"Noting that after merge, needs a subsequent quibble release and rebuild of images for CI jobs before it's active."

Now that https://gerrit.wikimedia.org/r/1296700 is merged/deployed this is active. Quibble will retry any time 1 time if a GnuTLS error is the cause of a git clone failure on the jobs listed on that patchset (not every job, but a bunch of jobs).

WMDE-Fisch awarded a token.Jun 3 2026, 6:35 AM

dancy awarded a token.Jun 3 2026, 2:42 PM

CDanis subscribed.Jun 4 2026, 3:04 PM

The patch has been merged for a week now, there has been an inflection in the trend since June 3rd:

The daily GnuTLS failure rate has visibly decreased. Lowering the priority of that task as it has now a smaller impact on our builds.
The remaining bits will require more debugging to be 100% fixed.

I'm still seeing a bunch of Recv failures / connection resets: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/41139/console . I've tried to merge 5 patches this morning and hit this issue on gate-and-submit for every single one of them.

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/41145/console

KineticPelagic subscribed.Jun 16 2026, 9:50 AM

ABran-WMF changed the task status from Open to In Progress.Jun 16 2026, 12:42 PM

ABran-WMF moved this task from Work in Progress (Tracking tasks) to Work in Progress on the Collaboration-Services board.

Change #1302819 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[integration/quibble@master] zuul: retry on connection resets, not only GnuTLS errors

https://gerrit.wikimedia.org/r/1302819

Change #1302829 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] ci: monitor a wider variety of network errors

https://gerrit.wikimedia.org/r/1302829

Change #1302834 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] ci: add repositories to gitcache

https://gerrit.wikimedia.org/r/1302834

Nemoralis unsubscribed.Jun 17 2026, 3:07 AM

ABran-WMF changed the task status from In Progress to Open.Jun 17 2026, 6:51 AM

ABran-WMF claimed this task.

ABran-WMF moved this task from Work in Progress to Awaiting Input on the Collaboration-Services board.

Change #1302829 merged by Arnaudb:

[operations/puppet@production] ci: monitor a wider variety of network errors

https://gerrit.wikimedia.org/r/1302829

Change #1303344 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] ci: update jenkins build monitor

https://gerrit.wikimedia.org/r/1303344

Change #1303344 merged by Arnaudb:

[operations/puppet@production] ci: update jenkins build monitor

https://gerrit.wikimedia.org/r/1303344

In T420865#12022630, @ArthurTaylor wrote:

I'm still seeing a bunch of Recv failures / connection resets: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/41139/console . I've tried to merge 5 patches this morning and hit this issue on gate-and-submit for every single one of them.

(For search purposes: that’s now “Recv failure: Connection reset by peer”, a slightly different error message from the GnuTLS recv errors.)

Change #1302819 merged by jenkins-bot:

[integration/quibble@master] zuul: retry on connection resets, not only GnuTLS errors

https://gerrit.wikimedia.org/r/1302819

Mentioned in SAL (#wikimedia-releng) [2026-06-18T18:50:34Z] <dancy> Tag Quibble 1.18.2 @ 152b497d317eaafc3cec334d5ce7e549697a2980 # T420865

Change #1304162 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[integration/config@master] dockerfiles: update Quibble to 1.18.2

https://gerrit.wikimedia.org/r/1304162

Change #1304162 merged by jenkins-bot:

[integration/config@master] dockerfiles: update Quibble to 1.18.2

https://gerrit.wikimedia.org/r/1304162

Mentioned in SAL (#wikimedia-releng) [2026-06-18T19:52:52Z] <dancy> Building quibble 1.18.2 images on contint primary (T420865)

Change #1304181 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[integration/config@master] Upgrade Quibble jobs to 1.18.2

https://gerrit.wikimedia.org/r/1304181

Mentioned in SAL (#wikimedia-releng) [2026-06-19T06:13:41Z] <hashar> Upgrading Quibble jobs to 1.18.2 (retry git connections on reset) https://gerrit.wikimedia.org/r/c/integration/config/+/1304181 # T420865

Change #1304181 merged by jenkins-bot:

[integration/config@master] Upgrade Quibble jobs to 1.18.2

https://gerrit.wikimedia.org/r/1304181

Novem_Linguae unsubscribed.Jun 19 2026, 6:22 AM

Given the recent evolution, T429749: Gerrit: network errors in CI was created to simplify tracking of that issue.

ABran-WMF changed the status of subtask T429749: Gerrit: network errors in CI from Open to In Progress.Jun 22 2026, 7:21 AM

ABran-WMF mentioned this in T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos.Jun 22 2026, 3:44 PM

As a result of 9be946fb3cfef2aa9103dc08e2be3188368b72a3 , the underlying error is no more shown and thus not captured in the Jenkins build logs. I am pretty sure this is still occurring, it definitely happened to me last week.

I think last time we ruled out Gerrit/Apache/Envoy and the only things left are:

the CDN / Edge cache (which would be SRE traffic)
WMCS NAT? But that would be surprising.

@ABran-WMF can you please loop SRE Traffic so they can investigate what has been abruptly terminating connection since Gerrit moved behind the CDN? Thx!

In T420865#12046290, @hashar wrote:

@ABran-WMF can you please loop SRE Traffic so they can investigate what has been abruptly terminating connection since Gerrit moved behind the CDN? Thx!

Yes! it's already been done in the ticket highlighted in my previous comment:

In T420865#12039170, @ABran-WMF wrote:

Given the recent evolution, T429749: Gerrit: network errors in CI was created to simplify tracking of that issue.

I'll mark this one again as Resolved because the issue has evolved and is being tracked in T429749.

ABran-WMF closed this task as Resolved.Jun 24 2026, 6:06 AM

hashar mentioned this in T430627: Add more repositories to the CI instances gitcache mirrors.Jun 30 2026, 9:09 AM

Change #1302834 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] ci: add repositories to gitcache

https://gerrit.wikimedia.org/r/1302834

Change #1302834 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] ci: add repositories to gitcache

https://gerrit.wikimedia.org/r/1302834

Change #1302834 merged by Arnaudb:

[operations/puppet@production] ci: add repositories to gitcache

https://gerrit.wikimedia.org/r/1302834