Page MenuHomePhabricator

replication/gerrit2001 issues
Closed, ResolvedPublic

Description

I am unable to fetch from gerrit-replica nor replicate to gerrit2001:

$ ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/DiscussionTools --wait                                                                                                                 Replicate mediawiki/extensions/DiscussionTools ref ..all.. to github.com, Succeeded! (OK)
# (no further success or failure messages after ~20 minutes)

On https://phabricator.wikimedia.org/diffusion/EDTO/manage/:

Pull of 'rEDTO' failed: Command failed with error #128! COMMAND git ls-remote '********' 'refs/*' STDOUT (empty) STDERR fatal: remote error: mediawiki/extensions/DiscussionTools unavailable
Error updating working copy: Command failed with error #128!
COMMAND
git ls-remote '********' 'refs/*'

STDOUT
(empty)

STDERR
fatal: remote error: mediawiki/extensions/DiscussionTools unavailable

This didn't happened when I used https://gerrit.wikimedia.org/... instead of https://gerrit-replica.wikimedia.org/.... See https://phabricator.wikimedia.org/diffusion/EDTO/uri/view/23061/.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Oct 9, 9:42 PM
MarcoAurelio triaged this task as High priority.Wed, Oct 9, 9:42 PM
Dzahn added a subscriber: Dzahn.Wed, Oct 9, 9:47 PM

Broken by https://gerrit.wikimedia.org/r/c/operations/puppet/+/541386 when we renamed the replication target yesterday.

root cause: reject HostKey: gerrit-replica.wikimedia.org

as shown in replication.log

Change 541931 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "Gerrit: Switch replication url for replica to gerrit-replica"

https://gerrit.wikimedia.org/r/541931

Change 541931 merged by Dzahn:
[operations/puppet@production] Revert "Gerrit: Switch replication url for replica to gerrit-replica"

https://gerrit.wikimedia.org/r/541931

Mentioned in SAL (#wikimedia-operations) [2019-10-09T22:01:39Z] <mutante> restarting gerrit to revert replication config change (T235135)

Dzahn added a comment.Wed, Oct 9, 10:10 PM

replication.log shows it is replicating again and working on the backlog queue right now.

MarcoAurelio closed this task as Resolved.Wed, Oct 9, 10:41 PM
MarcoAurelio assigned this task to Dzahn.

It looks everything is back to normal now.