Change Details

Git repo replication between gerrit1003 and github.com was found to be broken in {T416912}. Things are assumed to have gotten out of sync because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 changing Gerrit's replication configuration. The resolution was restarting the gerrit service. >>! In T416912#11598660, @hashar wrote: > We have a dashboard at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream though it is hard to read and does not make the issue stand out. > > What I didn't caught earlier is that the replication configuration has been changed by @ABran-WMF at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 . He has removed the replication to the spare gerrit2003. > > The Gerrit replication has an issue somewhere, which causes it too loose track of replications when the config is changed. Even when a given replica config has not been altered :-( >>! In T416912#11598746, @hashar wrote: > @bd808 mentioned it to me over IRC. I have encountered the bug previously but honestly I never went to dig into the root cause or check whether upstream has a bug filed for it. Maybe that got fixed in a more recent version of Gerrit. > > Meanwhile this one has been fixed by restarting Gerrit. In the `::gerrit` module, the `"${gerrit_site}/etc/replication.config"` File resource could notify the `gerrit` `systemd::service` resource so that it restarts on config change. Or we should have some other mechanism to ensure that Gerrit is restarted following a replication config change.

Git repo replication between gerrit1003 and github.com was found to be broken in {T416912}. Things are assumed to have gotten out of sync because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 changing Gerrit's replication configuration. The resolution was restarting the gerrit service. >>! In T416912#11598660, @hashar wrote: > We have a dashboard at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream though it is hard to read and does not make the issue stand out. > > What I didn't caught earlier is that the replication configuration has been changed by @ABran-WMF at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 . He has removed the replication to the spare gerrit2003. > > The Gerrit replication has an issue somewhere, which causes it too loose track of replications when the config is changed. Even when a given replica config has not been altered :-( >>! In T416912#11598746, @hashar wrote: > @bd808 mentioned it to me over IRC. I have encountered the bug previously but honestly I never went to dig into the root cause or check whether upstream has a bug filed for it. Maybe that got fixed in a more recent version of Gerrit. > > Meanwhile this one has been fixed by restarting Gerrit. In the `::gerrit` module, the `"${gerrit_site}/etc/replication.config"` File resource could notify the `gerrit` `systemd::service` resource so that it restarts on config change. Or we could have some other mechanism to ensure that Gerrit is restarted following a replication config change.

Git repo replication between gerrit1003 and github.com was found to be broken in {T416912}. Things are assumed to have gotten out of sync because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 changing Gerrit's replication configuration. The resolution was restarting the gerrit service. >>! In T416912#11598660, @hashar wrote: > We have a dashboard at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream though it is hard to read and does not make the issue stand out. > > What I didn't caught earlier is that the replication configuration has been changed by @ABran-WMF at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1237450 . He has removed the replication to the spare gerrit2003. > > The Gerrit replication has an issue somewhere, which causes it too loose track of replications when the config is changed. Even when a given replica config has not been altered :-( >>! In T416912#11598746, @hashar wrote: > @bd808 mentioned it to me over IRC. I have encountered the bug previously but honestly I never went to dig into the root cause or check whether upstream has a bug filed for it. Maybe that got fixed in a more recent version of Gerrit. > > Meanwhile this one has been fixed by restarting Gerrit. In the `::gerrit` module, the `"${gerrit_site}/etc/replication.config"` File resource could notify the `gerrit` `systemd::service` resource so that it restarts on config change. Or we shcould have some other mechanism to ensure that Gerrit is restarted following a replication config change.