Page MenuHomePhabricator
Create Task
Maniphest T416929

Gerrit git replication should not break when Puppet changes its config
Open, LowPublicBUG REPORT
Actions

Description

Git repo replication between gerrit1003 and github.com was found to be broken in T416912: Replication to GitHub seems to have stalled. Things are assumed to have gotten out of sync because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 changing Gerrit's replication configuration. The resolution was restarting the gerrit service.

In T416912#11598660, @hashar wrote:

We have a dashboard at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream though it is hard to read and does not make the issue stand out.

What I didn't caught earlier is that the replication configuration has been changed by @ABran-WMF at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 . He has removed the replication to the spare gerrit2003.

The Gerrit replication has an issue somewhere, which causes it too loose track of replications when the config is changed. Even when a given replica config has not been altered :-(

In T416912#11598746, @hashar wrote:

@bd808 mentioned it to me over IRC. I have encountered the bug previously but honestly I never went to dig into the root cause or check whether upstream has a bug filed for it. Maybe that got fixed in a more recent version of Gerrit.

Meanwhile this one has been fixed by restarting Gerrit.

In the ::gerrit module, the "${gerrit_site}/etc/replication.config" File resource could notify the gerrit systemd::service resource so that it restarts on config change. Or we could have some other mechanism to ensure that Gerrit is restarted following a replication config change.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Revert^2 "Gerrit: Disable auto reloading replication config"Hasharoperations/puppetproduction+3 -1
Customize query in gerrit

Related Objects

Event Timeline

bd808 created this task.Feb 9 2026, 9:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2026, 9:29 PM
bd808 added a comment.Feb 9 2026, 9:57 PM

Now that I understand what I'm reading, https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Troubleshooting documents this problem as well:

Any changes to replication.config should be followed with a restart of the gerrit service to ensure that those changes are active. There is a bug with the replication plugin where it will reload the replication.config (and acknowledge that the configuration has been reloaded in the logs), but actually operate with a broken configuration in an unknown state until a restart.

If we can disarm this foot-gun we will probably be happier in the long run.

Reedy updated the task description. (Show Details)Feb 9 2026, 9:58 PM
gerritbot added a comment.Feb 9 2026, 10:03 PM

Change #1238043 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] Revert^2 "Gerrit: Disable auto reloading replication config"

https://gerrit.wikimedia.org/r/1238043

gerritbot added a project: Patch-For-Review.Feb 9 2026, 10:03 PM
• hashar added a comment.EditedFeb 9 2026, 10:04 PM

git log -GautoReload modules/gerrit has the history:

That got disable by Paladox in October 2019 referencing "a bug". I went to enable autoreloading in August 2022. I am pretty sure we had a similar issue some months ago but I probably did not file it as a task.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1238043 would disables the plugin autoreloading, I'll sync with @ABran-WMF .

Dzahn subscribed.Feb 9 2026, 10:07 PM

Or we should have some other mechanism to ensure that Gerrit is restarted following a replication config change.

Or we should do restarts (or replication config changes) during planned maintenance periods because it might log users out and triggers new monitoring we just implemented which creates alerts and additional tickets.

bd808 renamed this task from Puppet should restart Gerrit when changing it's replication config to Gerrit git replication should not break when Puppet changes its config.Feb 11 2026, 6:59 PM
bd808 updated the task description. (Show Details)
• ABran-WMF added a project: collaboration-services.Feb 12 2026, 7:25 AM
• hashar added a comment.Feb 23 2026, 8:27 AM

The short fix is to disable configuration autoreloading in the replication plugin.

It might have been fixed with Gerrit 3.11 (which was not backported to the 3.10 version we run). The changelog for the replication plugin states:

3982574 Fix FanoutConfigurationResource update API:

This fixes issue 351843807 where Gerrit changes were no longer syncing to Github.

And the commit message has:

When updating the FanoutConfigurationResource, the remote subsection needs to be removed as the subsection name is part of the config filename.

I guess we can have a follow up task to reenable the autoreloading that is tagged for Gerrit (Gerrit 3.11) .

gerritbot added a comment.Feb 25 2026, 6:46 PM

Change #1238043 merged by Dzahn:

[operations/puppet@production] Revert^2 "Gerrit: Disable auto reloading replication config"

https://gerrit.wikimedia.org/r/1238043

Dzahn added a comment.Feb 25 2026, 6:49 PM

The short fix is to disable configuration autoreloading in the replication plugin.

This config change has been merged. No (manual) Gerrit restart was performed so far to apply this config change itself.

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2026, 7:31 PM
• ABran-WMF triaged this task as Low priority.Mar 2 2026, 4:23 PM
• ABran-WMF moved this task from Incoming to Work in Progress (Tracking tasks) on the collaboration-services board.
LSobanski moved this task from Work in Progress (Tracking tasks) to Incoming on the collaboration-services board.Thu, Jun 18, 10:55 AM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.Mon, Jun 22, 3:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits