Beginning at 2015-05-21 15:47 Gerrit's replication logs are full of errors like

[2015-05-21 15:47:41,273] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to gerritslave@gallium.wikimedia.org:/srv/ssd/gerrit/mediawiki/extensions/ConfirmEdit.git
org.eclipse.jgit.errors.TransportException: gerritslave@gallium.wikimedia.org:/srv/ssd/gerrit/mediawiki/extensions/ConfirmEdit.git: Algorithm negotiation fail

Both the error message and the time it started hints at the latest
SSHD hardening getting in the way here.

The commits

• f73786e3afb199f0fcc7aa00544b7c44beda91c7
• 598389d08a95bfdb07148fa1a5f6708ed981ef2f

look relevant, as they got merged around that time, and they change
sshd config.

Judging from gerrit's logs, the affected replication targets are:

• antimony.wikimedia.org
• gallium.wikimedia.org
• lanthanum.eqiad.wmnet

Possible way's forward would be to

• teach Java/Jsch/Gerrit's replication plugin to connect using the settings that we use, or
• to backpaddle on the ssh hardening for the three affected hosts.

Since we WMF wants to get rid of gerrit, I am not too fond of fiddling
with Java/Jsch/Gerrit's replication plugin.

Could we instead leverage the fresh $disable_nist_kex, and
$explicit_macs on the three affected hosts?

Change 213216 had a related patch set uploaded (by QChris):
Turn off sshd MAC and KEX hardening for gerrit replication targets

https://gerrit.wikimedia.org/r/213216

Also branch on http://git.wikimedia.org/summary/operations%2Fpuppet for master needs to be changed to production or production needs to be changed to master please.

In T99990#1306479, @Paladox wrote:

Also branch on http://git.wikimedia.org/summary/operations%2Fpuppet for master needs to be changed to production or production needs to be changed to master please.

That's T52152: 404 for Gitblit's "Tree > HEAD" in operations/puppet repository

Change 213216 merged by Alexandros Kosiaris:
Turn off sshd MAC and KEX hardening for gerrit replication targets

https://gerrit.wikimedia.org/r/213216

How long would it take for gerrit and git to pickup the change so that they both can start working again.

Keep in mind also that Gerrit's not going to start replicating a repo again until the objects in it start changing (new commits, etc). So repos that lag may take a little longer to decide to catch up.

A gerrit admin could kick off a replication job for all repos but I'm on vacation now so it won't be me :p

In T99990#1311842, @demon wrote:

Keep in mind also that Gerrit's not going to start replicating a repo again until the objects in it start changing [...]

If the replication plugin got restarted or so, that'd be true. But that's not the case here.

The replication plugin keeps failed pushes in its queue and retries them automatically. Gerrit had already caught up :-)

But meh. Starting forced replication nonetheless :-P

In T99990#1312092, @QChris wrote:

Starting forced replication nonetheless

That back-fired due to github overloading repo names (see T100409).
Fixed by forcing replication of only the affected repos again from their gerrit names.
Added a warning against --all to wikitech: https://wikitech.wikimedia.org/w/index.php?title=Gerrit&diff=160816&oldid=153784