Page MenuHomePhabricator

Creating a Patchdemo wiki for VisualEditor fails on missing ref from gerrit-replica
Closed, ResolvedPublicBUG REPORT

Description

@DLynch reports a recurring problem creating a wiki for this change:

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1311570

Still failing when I tried it a few minutes ago:

2026-07-16T22:36:20.903322080Z: ############### Applying requested patches ###############
2026-07-16T22:36:20.903762082Z: + cd /var/www/html/w/extensions/VisualEditor
2026-07-16T22:36:20.903822799Z: + git fetch origin refs/changes/70/1311570/1 master
2026-07-16T22:36:21.079743933Z: fatal: couldn't find remote ref refs/changes/70/1311570/1
Log stream has terminated, but deployment is not complete. Status is: starting

I wasn't able to reproduce this locally, but then remembered that we're using gerrit-replica, and sure enough:

🌒 16:42:20 brennen@inertia:~/visualeditor-replica ❀ git clone https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/VisualEditor.git
Cloning into 'VisualEditor'...
# successful clone here
🌒 16:42:34 brennen@inertia:~/visualeditor-replica ❀ cd VisualEditor
🌒 16:42:40 brennen@inertia:~/visualeditor-replica/VisualEditor (master =) ❀ git fetch origin refs/changes/70/1311570/1 master
fatal: couldn't find remote ref refs/changes/70/1311570/1
zsh: exit 128   git fetch origin refs/changes/70/1311570/1 master

Unclear if the replica is just lagging or if something else is going on here, filing a ticket for tracking just in case. I think this may have come up before but not entirely sure.

Event Timeline

Change #1311577 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] Revert "gerrit: add SSH config for replication between servers"

https://gerrit.wikimedia.org/r/1311577

Change #1311577 merged by Dzahn:

[operations/puppet@production] Revert "gerrit: add SSH config for replication between servers"

https://gerrit.wikimedia.org/r/1311577

Mentioned in SAL (#wikimedia-releng) [2026-07-16T23:06:56Z] <mutante> gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed T398401 but caused T432413 - replication is working again

brennen assigned this task to Dzahn.

Looks like this should be fixed now.

I wonder if it'd be worth having things fall back to the primary if something's up with the replica, although maybe that would just mask failures that need addressed anyway.

23:08 < mutante> [2026-07-16 23:08:09,697] Replication to gerrit@gerrit2002.wikimedia.org:/srv/gerrit/git/operations/puppet.git completed in 118065ms, 141080ms delay, 0 retries [CONTEXT 
                 pushOneId="89b5f8e6" ]

...

23:09 < mutante> repos/VisualEditor$ git fetch origin refs/changes/70/1311570/1 master
23:09 < mutante>  * branch                refs/changes/70/1311570/1 -> FETCH_HEAD
23:09 < brennen> that'll do it.
23:09 < mutante> yea, I freshly cloned it from the replica. confirmed