I've also updated the "restart gerrit" part: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Restarting

I've added a bit of documentation on gerrit ssh commands: https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Use_gerrit_ssh_commands

We should maybe have a prometheus metric exposed for 500 errors thrown from LDAP and/or failed syncs. That error looks transient because it recovers quickly (it was also the case this morning with T423674: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100), so it should maybe be caught/retried to generate an alert only if it lasts more than a few minutes/hours.

I have tried to limit max_concurrent_streams to 50, still inconclusive for the connection interruption in CI

a 500 error was thrown and the script crashed. It was a transient error

I have tried to increase per_connection_buffer_limit_bytes first to 4MB, then to 16MB to see if the stream watermark messages were a symptoms of the 502s. The message rate has reduced, but 502s are still happening (https://integration.wikimedia.org/ci/view/All/job/quibble-vendor-mysql-php83/lastFailedBuild/console). I'll keep on debugging.

quibble-with-gated-extensions-vendor-mysql-php83/27335 also failed during the clone phase with the same error. The failure happens in prepareRepo() / repo.prune() / git remote prune --dry-run origin:

Verbose logging has been enabled on Envoy on gerrit2003:

curl -i -X POST 'http://localhost:9631/logging?paths=http:debug,router:debug,connection:debug'

linked to T423156: setup phab1006

We might want to stop puppet first and have a merged change that will update httpd's config

In T333143#11817082, @Dzahn wrote:

Suggested run book to move remaining data:

prep:

• mkdir /srv/gerrit/site_path (we need/should use some new directory and this is what that is, /var/lib/gerrit (or previously /var/lib/gerrit2 is the default Gerrit Site Path.

• pre-rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/

migrate:

• stop gerrit

• rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/ one more time

• mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup (just in case, but don't forget it forever, /var/lib/gerrit needs to move out of the way though, or we simply "mv /var/lib/gerrit /srv/gerrit/site_path" and forget rsync!?)
• mount --bind /srv/gerrit/site_path /var/lib/gerrit (alternative: ln -s /srv/gerrit/site_path /var/lib/gerrit)

• start gerrit

?

No impact from that change over the past few hours:

root@contint1002:/srv/jenkins/builds# for i in $(fdfind --type file --changed-within 2h '^log$') ; do rg "GnuTLS recv error" -zli "${i}" -q && echo ${i} ; done|wc -l
7

I will keep on digging

Envoy now stops reusing connections to httpd on our Gerrit primary instance. I'll monitor the impact on the GnuTLS recv error rate.

A first manual test on gerrit-spare shows no breakage, I will now try to apply that change manually on the primary Gerrit instance before merging.

Transactions:		      408    hits
Availability:		      100.00 %

In T420623#11803372, @jcrespo wrote:

CC @ABran-WMF @Jelto that backups from gerrit & gitlab (and attempted recoveries) will be unavailable for that 2 hour window.

Thanks @DLynch @ArthurTaylor for reporting these builds.

In T421827#11798247, @ArthurTaylor wrote:

Got this just now: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/25784/console

Connection reuse has been disabled again. Please let me know if the 502 errors are still happening.

In T422070#11797717, @Jelto wrote:

I agree, we should align the permission and make it either private or public, like the other projects. I can reach out to Nat.

In T422070#11794677, @Jelto wrote:

I don't think this script should fail hard for a improperly configured project. I'll add proper error handling to the script which should fix broken package pulling.

alerts were stemming from esams maintenance.

thanks for highlighting this @taavi, the change has been merged to move these entries to state: production, feel free to reopen that issue if needed.

I forgot to add on my previous comment:

# for i in $(fdfind --type f --change-newer-than 2026-03-25 '^log$') ; do rg "GnuTLS recv error" -zli "${i}" -q && echo ${i} ; done|wc -l
114

There is a clear increase of these errors in the logs on contint, after we introduced Envoy and enabled connection-reuse.

In T421827#11785472, @A_smart_kitten wrote:

In T421827#11783908, @SomeRandomDeveloper wrote:

To be clear, this is basically just speculation based on the limited information I can find; but I am wondering about whether this type of error might potentially be unrelated to the addition of Envoy to Gerrit's stack.
Searching Gerrit for the text GnuTLS recv error (-54) (to try and find instances where this CI error has been pasted into a Gerrit comment), it seems like an early singular occurrence of this issue may have been on 2026-02-18, with occurrences starting regularly from 2026-03-16 onwards. IIUC from T420909#11752446, Envoy was added to Gerrit's stack on 2026-03-26.

In T421827#11789099, @hashar wrote:

It is not an issue with the software (git) being used on the client side (Jenkins) but with the infrastructure. Connections get aborted based on idle timeout, maximum duration or timeout to first response byte and there are four systems in the chain: ATS, Envoy, Apache, Gerrit.

I think that was resolved by @Dzahn

Thanks for the follow-up @SomeRandomDeveloper! Given the results of our recent tweaks on Envoy, it is a bit hard to debug further without more details.
Would it be possible to run that job with a bit more verbosity? Something like GIT_CURL_VERBOSE=1 GIT_TRACE_CURL=1 git $command -v.

@Dzahn closing that one because we removed these lines with T420909: gerrit: Add Envoy in Gerrit's stack

As mentioned in T421827: gerrit: Adapt timeouts to avoid 502 errors in CI jobs:

In T421827#11781312, @ABran-WMF wrote:

[...] Please let me know if you still see these 502 errors.

We need to update permissions on wmf-navigator, marking this Stalled until then

thanks for these @SomeRandomDeveloper @DLynch, I've merged a config update. Please let me know if you still see these 502 errors.

got it: 2026-04-02 07:44:44,661 ERROR [gitlab_package_puller.GitlabPackagePuller] Skipping project repos/projects/wmf-navigator after GitLab API error while preparing package fetch: 403: 403 Forbidden

agreed @Arnoldokoth it seems a permission on Gitlab might have been changed, preventing the script to access a repo. The error is thrown in after the Zotero line:

The change has been merged, please let us know if that does not fix the situation.

@ArthurTaylor @SomeRandomDeveloper is it OK for me to retry some of these jobs to test my change?

thanks for raising these, I'll check

I'm not sure this is still a required, I've updated the timers in 1266149. Please @hashar, let me know if that is no longer required, I'll drop the patch and mark this as Resolved

follow up done in T421736: Daily mediawiki-core-phpmetrics job broken since March 26, 2026 and T421827: gerrit: Adapt timeouts to avoid 502 errors in CI jobs:
Envoy upstream timeout has been updated to allow longer git fetch commands, let me know if that issue is still present after that.

after merging that config change, mediawiki-core-phpmetrics (from T421736: Daily mediawiki-core-phpmetrics job broken since March 26, 2026) now shows a successful build, @Jdlrobson @DLynch please let me know if your issue is still present after that change.

@dancy I've updated Envoy's configuration to increase the upstream_response_timeout to 300s instead of 120 → I've triggered a mediawiki-core-phpmetrics build and it went OK: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/705/console

Envoy has been added to our stack, we need to follow up in T421827: gerrit: Adapt timeouts to avoid 502 errors in CI jobs to adapt configurations

thanks @dancy for highlighting this, that is probably stemming from T420909: gerrit: Add Envoy in Gerrit's stack.
I think operations/puppet/+/1262020 could fix it, wdyt @hashar?

In T246763#11765934, @dancy wrote:

Yes, you should have access to do that now.

The new training flow keeps the existing VRTS export unchanged: vrts.TicketExport2Mbox.pl still produces mbox input in /var/spool/spam/{spam,ham}, and SpamAssassin is still trained with sa-learn --mbox and spamassassin --add-to-blacklist/--add-to-whitelist --mbox.

In T246763#11759008, @dancy wrote:

Try now please.

we tweaked several knobs on httpd and Envoy and still have the same underlying issue, I think aligning Jetty with the rest of the timers could yield more results

In T420909#11756114, @hashar wrote:

profile::tlsproxy::envoy::upstream_tls: true
profile::tlsproxy::envoy::upstream_response_timeout: 120.0

We need the counterpart for downstream (Apache) which currently has a timeout of 122. Maybe:

profile::tlsproxy::envoy::downstream_idle_timeout: 110

In T246763#11754283, @dancy wrote:

Note: If you get a 404 when visiting the job URL, click the "Sign In" button on the top left of the page, then try again.

We moved all gerrit instances behind CDN, they all have the same DNS pattern

I've been able to test my change on Pontoon:

Hey @neriah, thanks for raising that issue. With {T420595} and  T420909: gerrit: Add Envoy in Gerrit's stack the situation should be improved.
I'm marking this task as resolved, please let us know if it does not, or it does not sufficiently, and feel free to reopen this task.

In T420909#11747537, @ABran-WMF wrote:

In T420909#11742909, @ABran-WMF wrote:

gerrit-spare now uses Envoy to expose its service to the CDN.

so does gerrit-replica