| hashar | |
| Apr 17 2025, 8:48 AM |
| F59278037: dashboard_replication_delay.png | |
| Apr 17 2025, 8:48 AM |
Looking at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream?orgId=1 , the github replication delay is at 11.6 days:
Gerrit was restarted on April 2nd and there is a langish replication that fails:
e00ee77d 08:44:22.744 08:43:22.744 (retry 20630) [ebc62400] push git@github.com:wikimedia/apps-ios-wikipedia [..all..]
Maybe that is the reason for the high reported delay?
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| gerrit: do not replicate apps/ | operations/puppet | production | +10 -4 |
apps-ios-wikipedia had the replication denied in May 2017:
commit 56a74f4e7943560046679cb7d52211ba1e2f7b19
Author: Chad Horohoe <chadh@wikimedia.org>
Date: Thu May 4 17:25:51 2017 +0000
Modify access rules
diff --git a/groups b/groups
index 89eca50e..4443ad50 100644
--- a/groups
+++ b/groups
@@ -1,3 +1,4 @@
# UUID Group Name
#
412ae9ad870090512ddc1d553b34179937c42616 apps-ios-wikipedia
+820a70a4a276bb79b029e06283093d9288008500 mediawiki-replication
diff --git a/project.config b/project.config
index add79ea4..d8467409 100644
--- a/project.config
+++ b/project.config
@@ -10,3 +10,4 @@
owner = group apps-ios-wikipedia
push = group apps-ios-wikipedia
exclusiveGroupPermissions = push label-Code-Review label-Verified submit
+ read = deny group mediawiki-replicationThat was then restored by @thcipriani back in 2019 with:
commit 13265d0b07254b6ba58defe48e8221e2c90cd663
Author: Tyler Cipriani <tcipriani@wikimedia.org>
Date: Wed Mar 20 18:35:37 2019 +0000
thcipriani: remove refs/* Read
Only remaining group is empty/archived mediawiki-replication
diff --git a/groups b/groups
index 4443ad50..89eca50e 100644
--- a/groups
+++ b/groups
@@ -1,4 +1,3 @@
# UUID Group Name
#
412ae9ad870090512ddc1d553b34179937c42616 apps-ios-wikipedia
-820a70a4a276bb79b029e06283093d9288008500 mediawiki-replication
diff --git a/project.config b/project.config
index d8467409..add79ea4 100644
--- a/project.config
+++ b/project.config
@@ -10,4 +10,3 @@
owner = group apps-ios-wikipedia
push = group apps-ios-wikipedia
exclusiveGroupPermissions = push label-Code-Review label-Verified submit
- read = deny group mediawiki-replicationNot sure why?
The Gerrit pushes to GitHub are rejected since there is no matching repository https://github.com/wikimedia/apps-ios-wikimedia
https://gerrit.wikimedia.org/r/admin/repos/apps/android/wikipedia,general does not get replicated even though there is https://github.com/wikimedia/apps-android-wikipedia .
I know we can deny mediawiki-replication group from reading but eventually I have rediscovered we have:
[remote "github"] projects = ^(?:(?!apps\\/android\\/).)*$ projects = ^(?:(?!apps\\/ios\\/).)*$
So apparently that prevents the replication of apps/android/wikipedia but does not for apps/ios/wikimedia??
That is a large rabbit hole again :/
The regex to exclude the projects do not seem to work since they get scheduled:
$ zgrep --color=always -P 'apps/(ios|android).*github' replication_log.2025-04-02.gz [2025-04-02 15:26:18,810] scheduling replication apps/android:[..all..] => git@github.com:wikimedia/apps-android [2025-04-02 15:26:19,145] scheduling replication apps/ios:[..all..] => git@github.com:wikimedia/apps-ios [2025-04-02 15:26:19,576] scheduling replication apps/ios/wikipedia:[..all..] => git@github.com:wikimedia/apps-ios-wikipedia [2025-04-02 15:26:19,577] scheduled apps/ios/wikipedia:[..all..] => [cf01c408] push git@github.com:wikimedia/apps-ios-wikipedia [..all..] to run after 15s [2025-04-02 15:26:49,106] scheduling replication apps/android:[..all..] => git@github.com:wikimedia/apps-android [2025-04-02 15:26:49,109] scheduling replication apps/android/java-mwapi:[..all..] => git@github.com:wikimedia/apps-android-java-mwapi [2025-04-02 15:26:49,109] scheduled apps/android/java-mwapi:[..all..] => [03981d6d] push git@github.com:wikimedia/apps-android-java-mwapi [..all..] to run after 15s [2025-04-02 15:26:49,110] scheduling replication apps/android/translate:[..all..] => git@github.com:wikimedia/apps-android-translate [2025-04-02 15:26:49,111] scheduled apps/android/translate:[..all..] => [83abad80] push git@github.com:wikimedia/apps-android-translate [..all..] to run after 15s [2025-04-02 15:26:49,112] scheduling replication apps/android/wikipedia:[..all..] => git@github.com:wikimedia/apps-android-wikipedia [2025-04-02 15:26:49,112] scheduled apps/android/wikipedia:[..all..] => [03933d5d] push git@github.com:wikimedia/apps-android-wikipedia [..all..] to run after 15s [2025-04-02 15:26:49,119] scheduling replication apps/ios:[..all..] => git@github.com:wikimedia/apps-ios [2025-04-02 15:26:49,121] scheduling replication apps/ios/commons:[..all..] => git@github.com:wikimedia/apps-ios-commons [2025-04-02 15:26:49,122] scheduled apps/ios/commons:[..all..] => [405e0b9e] push git@github.com:wikimedia/apps-ios-commons [..all..] to run after 15s [2025-04-02 15:26:49,123] scheduling replication apps/ios/wikipedia:[..all..] => git@github.com:wikimedia/apps-ios-wikipedia [2025-04-02 15:26:49,123] consolidated apps/ios/wikipedia:[..all..] => (retry 1) [cf01c408] push git@github.com:wikimedia/apps-ios-wikipedia [..all..] with an existing pending push
My best guess is that on Gerrit start all repositories are scheduled, and they get filtered out as they are processed.
Mentioned in SAL (#wikimedia-releng) [2025-04-17T10:15:56Z] <hashar> gerrit: reparented apps.git to All-Archived-Projects.git in order to BLOCK mediawiki-replication. I have also archived all subprojects # T392198
I have marked apps/ as a child of All-Archived-Projects which BLOCK mediawiki-replication. The apps/ios/wikimedia entry, that kept being retried, has vanished:
$ ssh -p 29418 gerrit.wikimedia.org replication list --detail --remote github Remote: github Url: git@github.com:wikimedia/${name} AuthGroup: mediawiki-replication Project: ^(?:(?!apps\/android\/).)*$ Project: ^(?:(?!apps\/ios\/).)*$ In Flight: 0 Pending: 0
Why was it scheduled despite Project: ^(?:(?!apps\/ios\/).)*$ ?. I have no idea, my guess is that it apps/ios/ is matched by ^(?:(?!apps\/android\/).)* and thus pass through. If I look at the java code:
public boolean matches(Project.NameKey name) { if (projectPatterns.isEmpty()) { return true; } String projectName = name.get(); for (String pattern : projectPatterns) { if (matchesPattern(projectName, pattern)) { return true; } } return false; } private boolean matchesPattern(String projectName, String pattern) { boolean match = false; switch (getPatternType(pattern)) { case REGEX: match = projectName.matches(pattern); break; case WILDCARD: match = projectName.startsWith(pattern.substring(0, pattern.length() - 1)); break; case EXACT_MATCH: match = projectName.equals(pattern); } return match; }
It takes each of the pattern and return true if any match. So my assumption is correct, apps/android/wikipedia is matched by ^(?:(?!apps\/ios\/).)*$ and is thus allowed them to be replicated. That got introduced by:
commit 97c170306b9145a6cc0735e5166832d9734ef3bc
Author: Tyler Cipriani <tcipriani@wikimedia.org>
Date: Mon Aug 5 15:40:38 2019 -0600
gerrit: replication: exclude some projects
This should exclude the "apps/android/*" and the "apps/ios/*" projects
from mirroring from gerrit to github.
Change-Id: I72b527548aa14901c022ac9d1fd14dea772faf00
diff --git a/hieradata/role/common/gerrit.yaml b/hieradata/role/common/gerrit.yaml
index a9296e9b281..fa5548bcd82 100644
--- a/hieradata/role/common/gerrit.yaml
+++ b/hieradata/role/common/gerrit.yaml
@@ -31,6 +31,9 @@ gerrit::jetty::replication:
createMissingRepositories: false
maxRetries: 50
replicatePermissions: false
+ projects:
+ - '^(?:(?!apps\/ios\/).)*$'
+ - '^(?:(?!apps\/android\/).)*$'
slaves:
url: 'gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/${name}.git'
mirror: trueI have sent a patch to upstream to clarify the doc https://gerrit-review.googlesource.com/c/plugins/replication/+/468701
Change #1137256 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/puppet@production] gerrit: do not replicate apps/
Change #1137256 merged by Arnaudb:
[operations/puppet@production] gerrit: do not replicate apps/
After I have restarted Gerrit with:
replication.config [remote "github"] projects = ^(?:(?!apps\\/).)*$
The replication plugin did not attempt to replicate to GitHub any of the apps/* repos:
$ grep ' apps.*github' /var/log/gerrit/replication_log $
But still replicates to the Gerrit replica (expected):
[2025-04-17 14:14:45,292] scheduling replication apps/android/commons:[..all..] => gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps/android/commons.git [2025-04-17 14:14:45,292] scheduled apps/android/commons:[] => [6aeb1942] push gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps/android/commons.git [] to run after 5s [2025-04-17 14:15:14,856] scheduling replication apps:[..all..] => gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps.git [2025-04-17 14:15:14,856] scheduled apps:[..all..] => [5beab14f] push gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps.git [..all..] to run after 5s [2025-04-17 14:15:14,857] scheduling replication apps/android:[..all..] => gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps/android.git [2025-04-17 14:15:14,857] scheduled apps/android:[..all..] => [dbf54135] push gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps/android.git [..all..] to run after 5s [2025-04-17 14:15:14,858] scheduling replication apps/android/commons:[..all..] => gerrit2@gerrit1003.wikimedia.org:/srv/gerrit/git/apps/android/commons.git ...