Page MenuHomePhabricator
Create Task
Maniphest T395887

github mirror out of sync
Closed, ResolvedPublic
Actions

Description

Looks like after this last commit https://github.com/wikimedia/operations-puppet/commit/d66a3d24c5411ac8dccee6be72f8fc713d3843e2 our sync between gerrit and github has stopped:
https://github.com/wikimedia/operations-puppet/commits/production/

This seems to affect at least the puppet repo and the mediawiki-config one:
I pushed this to mw config repo: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1152893 but it is also not there: https://github.com/wikimedia/operations-mediawiki-config/commits/master/

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Revert^2 "gerrit: add a second replica, start replicating to gerrit2003"Dzahnoperations/puppetproduction+16 -8
Revert "gerrit: add a second replica, start replicating to gerrit2003"Hasharoperations/puppetproduction+8 -16
Customize query in gerrit

Related Objects

Event Timeline

Marostegui created this task.Jun 3 2025, 11:03 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJun 3 2025, 11:03 AM
Jelto added a subscriber: Dzahn.Jun 3 2025, 11:04 AM
A_smart_kitten subscribed.Jun 3 2025, 11:04 AM
Jelto subscribed.Jun 3 2025, 11:16 AM

CC from IRC:

Right after the last commit I found the error in Gerrit log "Error while deleting task f25ad588bb6cc6c345e1efca0b571c8bb262aad3 [CONTEXT PLUGIN="replication" project="operations/puppet" pushOneId="f14a5f1f" request="REST /r/*/operations%2Fpuppet~1140520/*/6/*" ]"
https://logstash.wikimedia.org/goto/85800e47347d6682c4948f13457b43af

SomeRandomDeveloper subscribed.Jun 3 2025, 11:32 AM

The repos of MW Core and various extensions I checked are also affected.

• ABran-WMF subscribed.Jun 3 2025, 12:26 PM
• hashar claimed this task.Jun 3 2025, 2:36 PM
• hashar subscribed.

The replicas are around:

$ ssh -p 29418 hashar@gerrit.wikimedia.org replication list
Remote: github
Url: git@github.com:wikimedia/${name}

Remote: replica-a-codfw
Url: gerrit2@gerrit2002.wikimedia.org:/srv/gerrit/git/${name}.git

Remote: replica-b-codfw
Url: gerrit2@gerrit2003.wikimedia.org:/srv/gerrit/git/${name}.git

The replications items toward github are in the file based queue:

$ grep -R github /var/lib/gerrit2/review_site/data/replication/ref-updates
/var/lib/gerrit2/review_site/data/replication/ref-updates/waiting/96308b8af4e8325697b48a50eb34bb65fad33b5e:{"project":"phabricator","refs":["..all.."],"uri":"git@github.com:wikimedia/phabricator","remote":"github"}
/var/lib/gerrit2/review_site/data/replication/ref-updates/waiting/05977aba9cfd34ac4f02a6d11410924adaac4405:{"project":"wikimedia/fundraising/crm/modules","refs":["..all.."],"uri":"git@github.com:wikimedia/wikimedia-fundraising-crm-modules","remote":"github"}
/var/lib/gerrit2/review_site/data/replication/ref-updates/waiting/5db191ab1eba201e007a7d71a1e467a1b4227eb5:{"project":"wikidata","refs":["..all.."],"uri":"git@github.com:wikimedia/wikidata","remote":"github"}
/var/lib/gerrit2/review_site/data/replication/ref-updates/waiting/fc2835dd64815484eae0c7985c3636dcea14c4a8:{"project":"research","refs":["..all.."],"uri":"git@github.com:wikimedia/research","remote":"github"}

The reason is a new replica was added and:

  1. the new host (gerrit2003) does not work
[2025-06-03 14:35:07,262] Cannot replicate to gerrit2@gerrit2003.wikimedia.org:/srv/gerrit/git/mediawiki/extensions/BlueSpiceConfigManager.git [CONTEXT pushOneId="2f7e55b8" ]
org.eclipse.jgit.errors.TransportException: gerrit2@gerrit2003.wikimedia.org:/srv/gerrit/git/mediawiki/extensions/BlueSpiceConfigManager.git: Cannot log in at gerrit2003.wikimedia.org:22
publickey: trying ssh-rsa key SHA256:ZXTN1Amy5Vlu14w7uwmpe8Hj/JCXOrlnOaDzmWwFbd8 with signature type rsa-sha2-512
publickey: trying ssh-rsa key SHA256:ZXTN1Amy5Vlu14w7uwmpe8Hj/JCXOrlnOaDzmWwFbd8 with signature type rsa-sha2-256
publickey: trying ssh-rsa key SHA256:ZXTN1Amy5Vlu14w7uwmpe8Hj/JCXOrlnOaDzmWwFbd8 with signature type ssh-rsa
publickey: no more keys to try
  1. Gerrit was not restarted ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/1140520/comments/0fac314f_6ba08164 ) so it got confused when reloading the config.
gerritbot added a comment.Jun 3 2025, 2:43 PM

Change #1153159 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] Revert "gerrit: add a second replica, start replicating to gerrit2003"

https://gerrit.wikimedia.org/r/1153159

gerritbot added a project: Patch-For-Review.Jun 3 2025, 2:43 PM
gerritbot added a comment.Jun 3 2025, 2:52 PM

Change #1153159 merged by AOkoth:

[operations/puppet@production] Revert "gerrit: add a second replica, start replicating to gerrit2003"

https://gerrit.wikimedia.org/r/1153159

• hashar added a comment.Jun 3 2025, 3:02 PM

That broke replication (T395887) because:

  • gerrit2003 uses a different username (T338470 should be completed)
  • its ssh host key was not known to the gerrit2 user on the primary
  • and most probably we must restart Gerrit on changes made to the replica config

For the later I am not sure, cause immediately after having deployed the change, the config got taken in account:

before
$ ssh -p 29418 gerrit.wikimedia.org replication list
Remote: github
Url: git@github.com:wikimedia/${name}

Remote: replica-a-codfw
Url: gerrit2@gerrit2002.wikimedia.org:/srv/gerrit/git/${name}.git

Remote: replica-b-codfw
Url: gerrit2@gerrit2003.wikimedia.org:/srv/gerrit/git/${name}.git

I ran Puppet and:

after
$ ssh -p 29418 hashar@gerrit.wikimedia.org replication list
Remote: github
Url: git@github.com:wikimedia/${name}

Remote: replica_codfw
Url: gerrit2@gerrit2002.wikimedia.org:/srv/gerrit/git/${name}.git

And the items in /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting are not magically retrieved. My assumption is that on reconfiguration Gerrit loose track of all those entries cause the in memory objects are discard (while the files are left behind).

I will restart Gerrit to ensure everything is properly set.

• hashar mentioned this in T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Jun 3 2025, 3:04 PM
Stashbot added a comment.Jun 3 2025, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-03T15:06:18Z] <hashar> Restarted Gerrit due to issue with replication config | T395887

• hashar closed this task as Resolved.Jun 3 2025, 3:11 PM

I have seen replication pass through so I guess it is fixed. The replication is going on right now, it will eventually reaches operations/puppet at some point :)

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2025, 3:30 PM
gerritbot added a comment.Jun 4 2025, 7:08 PM

Change #1153265 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] Revert^2 "gerrit: add a second replica, start replicating to gerrit2003"

https://gerrit.wikimedia.org/r/1153265

gerritbot added a project: Patch-For-Review.Jun 4 2025, 7:08 PM
• thcipriani added a project: Essential-Work.Jun 5 2025, 11:59 PM
• thcipriani added a project: Release-Engineering-Team (Doing 😎).
gerritbot added a comment.Jul 7 2025, 5:11 PM

Change #1153265 merged by Dzahn:

[operations/puppet@production] Revert^2 "gerrit: add a second replica, start replicating to gerrit2003"

https://gerrit.wikimedia.org/r/1153265

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2025, 5:30 PM
Dzahn added a comment.Jul 7 2025, 6:00 PM

We activated replication to the new machine again and unified replication config to use the same settings for both hosts.

A service restart has happened and github replication also looked fine this time.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits