Page MenuHomePhabricator

replication/gerrit2001 issues
Closed, ResolvedPublic

Description

I am unable to fetch from gerrit-replica nor replicate to gerrit2001:

$ ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/DiscussionTools --wait                                                                                                                 Replicate mediawiki/extensions/DiscussionTools ref ..all.. to github.com, Succeeded! (OK)
# (no further success or failure messages after ~20 minutes)

On https://phabricator.wikimedia.org/diffusion/EDTO/manage/:

Pull of 'rEDTO' failed: Command failed with error #128! COMMAND git ls-remote '********' 'refs/*' STDOUT (empty) STDERR fatal: remote error: mediawiki/extensions/DiscussionTools unavailable
Error updating working copy: Command failed with error #128!
COMMAND
git ls-remote '********' 'refs/*'

STDOUT
(empty)

STDERR
fatal: remote error: mediawiki/extensions/DiscussionTools unavailable

This didn't happened when I used https://gerrit.wikimedia.org/... instead of https://gerrit-replica.wikimedia.org/.... See https://phabricator.wikimedia.org/diffusion/EDTO/uri/view/23061/.

Event Timeline

Broken by https://gerrit.wikimedia.org/r/c/operations/puppet/+/541386 when we renamed the replication target yesterday.

root cause: reject HostKey: gerrit-replica.wikimedia.org

as shown in replication.log

Change 541931 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "Gerrit: Switch replication url for replica to gerrit-replica"

https://gerrit.wikimedia.org/r/541931

Change 541931 merged by Dzahn:
[operations/puppet@production] Revert "Gerrit: Switch replication url for replica to gerrit-replica"

https://gerrit.wikimedia.org/r/541931

Mentioned in SAL (#wikimedia-operations) [2019-10-09T22:01:39Z] <mutante> restarting gerrit to revert replication config change (T235135)

replication.log shows it is replicating again and working on the backlog queue right now.

MarcoAurelio assigned this task to Dzahn.

It looks everything is back to normal now.