Page MenuHomePhabricator
Create Task
Maniphest T418108

gerrit: move gerrit-replica behind CDN
Closed, ResolvedPublic
Actions

Description

We want to move all gerrit instances behind CDN, that will:

Whats to be discussed:

  • Is the replica traffic going over tcp-proxies and gerrit-lb or directly to the CDN?
  • what about SSH traffic? I guess the replicas don't need working SSH
    • is replication is happening over ssh on 29418?

Whats needed:

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
gerrit: dns cache wipe updateArnaudboperations/cookbooksmaster+2 -2
Revert^2 "gerrit: move gerrit-replica behind CDN"Arnaudboperations/dnsmaster+2 -5
gerrit: move gerrit-replica behind CDNArnaudboperations/dnsmaster+2 -5
gerrit: move gerrit-replica behind CDNArnaudboperations/dnsmaster+1 -4
cache:text: add gerrit-replica to alternate_domainsArnaudboperations/puppetproduction+2 -0
cache:text: add gerrit-replica to alternate_domainsJeltooperations/puppetproduction+2 -0
trafficserver: Add gerrit-replica backendArnaudboperations/puppetproduction+8 -0
gerrit: add gerrit-replica service to LVSArnaudboperations/puppetproduction+35 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• ABran-WMF created this task.Feb 23 2026, 10:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2026, 10:07 AM
• ABran-WMF changed the task status from Open to In Progress.Feb 23 2026, 10:07 AM
• ABran-WMF triaged this task as High priority.
• ABran-WMF added a subtask: T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use.
• ABran-WMF changed the status of subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use from Open to Stalled.
• ABran-WMF mentioned this in T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use.
• ABran-WMF moved this task from Incoming to Work in Progress on the collaboration-services board.
gerritbot added a comment.Feb 23 2026, 2:59 PM

Change #1240294 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: add gerrit-replica service to LVS

https://gerrit.wikimedia.org/r/1240294

gerritbot added a project: Patch-For-Review.Feb 23 2026, 2:59 PM

Change #1240603 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: add gerrit-replica backend to LVS

https://gerrit.wikimedia.org/r/1240603

• ABran-WMF updated the task description. (Show Details)Feb 23 2026, 3:01 PM
• ABran-WMF added a parent task: T417536: ATS causes git fetches from Gerrit to fail with 502 responses.
• ABran-WMF created subtask T418361: gerrit: move gerrit-spare behind CDN.Feb 25 2026, 11:09 AM
• ABran-WMF mentioned this in T418361: gerrit: move gerrit-spare behind CDN.
Novem_Linguae renamed this task from gerrit: gerrit-replica behind CDN to gerrit: move gerrit-replica behind CDN.Feb 27 2026, 10:25 AM
Jelto added a subtask: Restricted Task.Feb 27 2026, 10:32 AM
Jelto updated the task description. (Show Details)Feb 27 2026, 10:47 AM
gerritbot added a comment.Feb 27 2026, 10:48 AM

Change #1240294 abandoned by Jelto:

[operations/puppet@production] gerrit: add gerrit-replica service to LVS

Reason:

not needed

https://gerrit.wikimedia.org/r/1240294

gerritbot added a comment.Mar 2 2026, 10:16 AM

Change #1240603 merged by Jelto:

[operations/puppet@production] trafficserver: Add gerrit-replica backend

https://gerrit.wikimedia.org/r/1240603

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2026, 10:30 AM
• ABran-WMF updated the task description. (Show Details)Mar 2 2026, 10:39 AM
gerritbot added a comment.Mar 2 2026, 10:59 AM

Change #1247026 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] cache:text: add gerrit-replica to alternate_domains

https://gerrit.wikimedia.org/r/1247026

gerritbot added a project: Patch-For-Review.Mar 2 2026, 10:59 AM
• ABran-WMF added a comment.EditedMar 2 2026, 11:04 AM

Is the replica traffic going over tcp-proxies and gerrit-lb or directly to the CDN?

The replication traffic will stay the same, using the hosts FQDN

what about SSH traffic? I guess the replicas don't need working SSH

We decided to not expose the 29418 SSH port for the replica and spare.

is replication is happening over ssh on 29418?

Replication is going through ssh port on tcp/22 and uses the system user gerrit:

[remote "replica"]
[...]
  url = gerrit@gerrit1003.wikimedia.org:/srv/gerrit/git/${name}.git
• ABran-WMF updated the task description. (Show Details)Mar 2 2026, 11:05 AM
gerritbot added a comment.Mar 3 2026, 9:00 AM

Change #1247530 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: move gerrit-replica behind CDN

https://gerrit.wikimedia.org/r/1247530

• ABran-WMF updated the task description. (Show Details)Mar 3 2026, 9:00 AM
• ABran-WMF updated the task description. (Show Details)Mar 3 2026, 9:06 AM
• ABran-WMF updated the task description. (Show Details)
• ABran-WMF changed the status of subtask T418361: gerrit: move gerrit-spare behind CDN from Open to In Progress.Mar 3 2026, 9:14 AM
gerritbot added a comment.Mar 3 2026, 9:22 AM

Change #1247534 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: dns cache wipe update

https://gerrit.wikimedia.org/r/1247534

• ABran-WMF updated the task description. (Show Details)Mar 3 2026, 9:54 AM
gerritbot added a comment.Mar 4 2026, 9:28 AM

Change #1247026 merged by Jelto:

[operations/puppet@production] cache:text: add gerrit-replica to alternate_domains

https://gerrit.wikimedia.org/r/1247026

gerritbot added a comment.Mar 4 2026, 9:45 AM

Change #1247940 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] cache:text: add gerrit-replica to alternate_domains

https://gerrit.wikimedia.org/r/1247940

gerritbot added a comment.Mar 4 2026, 9:48 AM

Change #1247940 merged by Arnaudb:

[operations/puppet@production] cache:text: add gerrit-replica to alternate_domains

https://gerrit.wikimedia.org/r/1247940

• ABran-WMF updated the task description. (Show Details)Mar 4 2026, 9:49 AM
gerritbot added a comment.Mar 4 2026, 10:48 AM

Change #1247964 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: move gerrit-replica behind CDN

https://gerrit.wikimedia.org/r/1247964

gerritbot added a comment.Mar 4 2026, 10:48 AM

Change #1247530 abandoned by Arnaudb:

[operations/dns@master] gerrit: move gerrit-replica behind CDN

Reason:

merge conflict: replaced by 1247964

https://gerrit.wikimedia.org/r/1247530

• ABran-WMF updated the task description. (Show Details)Mar 4 2026, 10:52 AM
• ABran-WMF closed subtask T418361: gerrit: move gerrit-spare behind CDN as Resolved.Mar 4 2026, 10:54 AM
gerritbot added a comment.Mar 4 2026, 1:02 PM

Change #1247964 merged by Arnaudb:

[operations/dns@master] gerrit: move gerrit-replica behind CDN

https://gerrit.wikimedia.org/r/1247964

• ABran-WMF updated the task description. (Show Details)Mar 4 2026, 1:04 PM
• ABran-WMF closed this task as Resolved.Mar 4 2026, 1:11 PM

gerrit-replica has been moved behind CDN, everything looks OK

Jelto subscribed.Mar 4 2026, 1:25 PM

Thank you @ABran-WMF for merging the DNS change. I can confirm gerrit-replica host is accessible over the CDN now:

curl -I https://gerrit-replica.wikimedia.org/r/plugins/gitiles/operations/puppet/
HTTP/2 200 
date: Wed, 04 Mar 2026 13:24:31 GMT
x-cache: cp3072 miss, cp3072 pass

And there are already request in superset:

https://superset.wikimedia.org/superset/dashboard/p/xn8BnoRWrzY/

Jelto awarded a token.Mar 4 2026, 1:25 PM
Jelto changed the status of subtask Restricted Task from Stalled to Open.
• ABran-WMF reopened this task as Open.Mar 4 2026, 1:33 PM
gerritbot added a comment.Mar 4 2026, 1:41 PM

Change #1248002 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^2 "gerrit: move gerrit-replica behind CDN"

https://gerrit.wikimedia.org/r/1248002

• ABran-WMF updated the task description. (Show Details)Mar 4 2026, 1:41 PM
• ABran-WMF mentioned this in T419009: Zuul has lost connection to gerrit, freezing all CI in queue-only mode.
• ABran-WMF added a subscriber: • hashar.Mar 4 2026, 2:08 PM

the change to put gerrit-replica behind CDN has been reverted to see if it triggered T419009: Zuul has lost connection to gerrit, freezing all CI in queue-only mode, we confirmed with @hashar that it was apparently not related

gerritbot added a comment.Mar 4 2026, 2:26 PM

Change #1248002 merged by Arnaudb:

[operations/dns@master] Revert^2 "gerrit: move gerrit-replica behind CDN"

https://gerrit.wikimedia.org/r/1248002

• ABran-WMF closed this task as Resolved.Mar 4 2026, 2:37 PM
• ABran-WMF updated the task description. (Show Details)

gerrit-replica is back again behind CDN, was not the trigger for T419009: Zuul has lost connection to gerrit, freezing all CI in queue-only mode

• ABran-WMF changed the status of subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use from Stalled to In Progress.Mar 5 2026, 1:33 PM
Jelto closed subtask Restricted Task as Resolved.Mar 10 2026, 10:33 AM
• ABran-WMF closed subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use as Resolved.Mar 16 2026, 9:24 AM
• ABran-WMF reopened subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use as Open.Mar 16 2026, 9:49 AM
• ABran-WMF closed subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use as Resolved.Mar 16 2026, 1:49 PM
gerritbot added a comment.Mar 17 2026, 12:19 PM

Change #1247534 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: dns cache wipe update

https://gerrit.wikimedia.org/r/1247534

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2026, 12:32 PM
• hashar reopened subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use as Stalled.Mar 18 2026, 8:55 AM
• ABran-WMF closed subtask T417998: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use as Resolved.Mar 23 2026, 1:17 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits