Page MenuHomePhabricator
Create Task
Maniphest T408532

Deploy a TCP proxy across all DCs
Closed, ResolvedPublic
Actions

Description

TCP-proxy specific work for {T365259}

Active patches:

topic branch: https://gerrit.wikimedia.org/r/q/topic:%22tcp-proxy%22

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: change healthcheck URL for service catalogoperations/puppetproduction+1 -2
tcpproxy: switch firewall provider from nftables to fermoperations/puppetproduction+1 -1
Add Cumin alias for tcpproxy hostsoperations/puppetproduction+4 -0
tcpproxy: include profile::lvs::realserver in roleoperations/puppetproduction+5 -0
service: add gerrit-https service to service catalogoperations/puppetproduction+62 -0
git_ssh_proxy: add role::git_ssh_proxy for Gerrit and GitLab ssh proxiesoperations/puppetproduction+120 -1
allocate codfw VIP for load-balanced tcp-proxy serviceoperations/dnsmaster+2 -1
allocate eqiad VIP for load balanced tcp-proxy serviceoperations/dnsmaster+2 -1
tcpproxy: allow PRODUCTION_NETWORKS to connect to 29418operations/puppetproduction+1 -1
tcpproxy: haproxy: log level change to infooperations/puppetproduction+1 -1
tcpproxy: haproxy: listen on v4+v6 for both portsoperations/puppetproduction+2 -3
tcpproxy: haproxy: make stats workoperations/puppetproduction+1 -0
tcpproxy: add simple puppet service resource to manage haproxyoperations/puppetproduction+6 -0
tcpproxy: use notify to ensure service gets restarted on config changesoperations/puppetproduction+1 -0
gerrit: allow production networks to connect to gerrit-sshoperations/puppetproduction+2 -2
prometheus::ops: add tcpproxies scrapeoperations/puppetproduction+20 -2
tcpproxy: greatly reduce connection timeoutsoperations/puppetproduction+3 -3
site: apply tcpproxy role on all VMs created for itoperations/puppetproduction+2 -7
tcpproxy: add basic logging configoperations/puppetproduction+3 -1
tcpproxy: add firewall rule to allow gerrit ssh portoperations/puppetproduction+5 -0
tcpproxy: add config template and parametersoperations/puppetproduction+55 -1
tcpproxy: set puppet7 and firewall provider to ferm for new roleoperations/puppetproduction+2 -0
site: fix regex for tcp-proxy to cover 1002operations/puppetproduction+1 -1
site/role: create placeholder role/profile for tcpproxyoperations/puppetproduction+21 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedDzahnT411895 gerrit behind CDN
 ResolvedDzahnT408532 Deploy a TCP proxy across all DCs
 ResolvedDzahnT408064 Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
LSobanski triaged this task as High priority.Oct 28 2025, 9:43 AM
LSobanski updated the task description. (Show Details)
LSobanski added a subtask: T408064: Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy).
LSobanski removed a project: WMF-NDA.Oct 28 2025, 1:35 PM
gerritbot added a comment.Oct 28 2025, 3:54 PM

Change #1198281 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] git_ssh_proxy: add role::git_ssh_proxy for Gerrit and GitLab ssh proxies

https://gerrit.wikimedia.org/r/1198281

gerritbot added a project: Patch-For-Review.Oct 28 2025, 3:54 PM
gerritbot added a comment.Oct 28 2025, 6:14 PM

Change #1198397 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/role: create placeholder role/profile for tcpproxy

https://gerrit.wikimedia.org/r/1198397

Dzahn moved this task from Incoming to Work in Progress (Tracking tasks) on the collaboration-services board.Oct 28 2025, 11:16 PM
Dzahn closed subtask T408064: Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy) as Resolved.Oct 29 2025, 9:26 PM
Dzahn added a comment.Oct 30 2025, 8:45 PM

config example kindly provided by Chris Danis:

P84490 tcp-proxy config
1<%#- SPDX-License-Identifier: Apache-2.0 -%>
2# Note: This file is managed by puppet.
3global
4 user haproxy
5 group haproxy
6 <%#- @socket is a path to a UNIX domain socket, which speaks this protocol
7 https://docs.haproxy.org/2.8/management.html#9.3 -%>
8 stats socket <%= @socket %> mode 600 level admin
9 hard-stop-after 5m
10 set-dumpable
11
12defaults
13 mode tcp
14 option dontlognull
15 option tcplog
16 option tcp-check
17 retries 1
18 timeout connect 50000ms
19 timeout client 500000ms
20 timeout server 500000ms
21
22<%#- TODO: for l in @listeners, etc etc. -%>
23listen gerrit_ssh
24 bind *:29418
25 server backend_server gerrit.discovery.wmnet port 29418 resolvers default init-addr none check maxconn 200 <%#- TODO: maxconn might be too low? %>
26
27<%- if @prometheus_port -%>
28frontend stats
29 no log
30 maxconn 100
31 bind :<%= @prometheus_port %>
32 bind :::<%= @prometheus_port %> v6only
33 http-request use-service prometheus-exporter if { path /metrics }
34 stats enable
35 stats uri /stats
36 stats refresh 10s
37 # Explicitly avoid keep-alive to prevent Prometheus scrapers from
38 # reusing indefinitelly the same TCP connection. See T343000
39 http-after-response set-header Connection Close
40<%- end -%>

gerritbot added a comment.Oct 30 2025, 8:47 PM

Change #1198397 merged by Dzahn:

[operations/puppet@production] site/role: create placeholder role/profile for tcpproxy

https://gerrit.wikimedia.org/r/1198397

gerritbot added a comment.Oct 30 2025, 11:30 PM

Change #1200188 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: fix regex for tcp-proxy to cover 1002

https://gerrit.wikimedia.org/r/1200188

gerritbot added a comment.Oct 30 2025, 11:31 PM

Change #1200188 merged by Dzahn:

[operations/puppet@production] site: fix regex for tcp-proxy to cover 1002

https://gerrit.wikimedia.org/r/1200188

gerritbot added a comment.Oct 30 2025, 11:36 PM

Change #1200189 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: set puppet7 and firewall provider to ferm for new role

https://gerrit.wikimedia.org/r/1200189

gerritbot added a comment.Oct 30 2025, 11:38 PM

Change #1200189 merged by Dzahn:

[operations/puppet@production] tcpproxy: set puppet7 and firewall provider to ferm for new role

https://gerrit.wikimedia.org/r/1200189

Stashbot added a comment.Oct 30 2025, 11:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-30T23:48:48Z] <mutante> forward-fixing to puppet7 on tcp-proxy1001/1002 per T349619 T408532

Stashbot mentioned this in T349619: Migrate roles to puppet7.Oct 30 2025, 11:48 PM
gerritbot added a comment.Oct 30 2025, 11:57 PM

Change #1200190 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: add config template

https://gerrit.wikimedia.org/r/1200190

gerritbot added a comment.Oct 31 2025, 12:50 AM

Change #1200190 merged by Dzahn:

[operations/puppet@production] tcpproxy: add config template and parameters

https://gerrit.wikimedia.org/r/1200190

taavi mentioned this in T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.Oct 31 2025, 11:56 AM
gerritbot added a comment.Nov 3 2025, 10:39 PM

Change #1201299 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: add firewall rule to allow gerrit ssh port

https://gerrit.wikimedia.org/r/1201299

gerritbot added a comment.Nov 3 2025, 10:50 PM

Change #1201299 merged by Dzahn:

[operations/puppet@production] tcpproxy: add firewall rule to allow gerrit ssh port

https://gerrit.wikimedia.org/r/1201299

gerritbot added a comment.Nov 4 2025, 12:26 AM

Change #1201311 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: add basic logging config

https://gerrit.wikimedia.org/r/1201311

gerritbot added a comment.Nov 4 2025, 12:32 AM

Change #1201312 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: apply tcpproxy role on all VMs created for it

https://gerrit.wikimedia.org/r/1201312

gerritbot added a comment.Nov 4 2025, 4:45 PM

Change #1201311 merged by Dzahn:

[operations/puppet@production] tcpproxy: add basic logging config

https://gerrit.wikimedia.org/r/1201311

gerritbot added a comment.Nov 4 2025, 6:01 PM

Change #1201745 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: greatly reduce connection timeouts

https://gerrit.wikimedia.org/r/1201745

Dzahn changed the task status from Open to In Progress.Nov 4 2025, 6:01 PM
Dzahn claimed this task.
gerritbot added a comment.Nov 4 2025, 6:59 PM

Change #1201745 merged by Dzahn:

[operations/puppet@production] tcpproxy: greatly reduce connection timeouts

https://gerrit.wikimedia.org/r/1201745

gerritbot added a comment.Nov 4 2025, 9:52 PM

Change #1201312 merged by Dzahn:

[operations/puppet@production] site: apply tcpproxy role on all VMs created for it

https://gerrit.wikimedia.org/r/1201312

gerritbot added a comment.Nov 4 2025, 10:12 PM

Change #1201810 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] prometheus::ops: add tcpproxies scrape

https://gerrit.wikimedia.org/r/1201810

gerritbot added a comment.Nov 4 2025, 10:24 PM

Change #1201810 merged by Dzahn:

[operations/puppet@production] prometheus::ops: add tcpproxies scrape

https://gerrit.wikimedia.org/r/1201810

gerritbot added a comment.Nov 4 2025, 10:29 PM

Change #1201820 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: allow production networks to connect to gerrit-ssh

https://gerrit.wikimedia.org/r/1201820

gerritbot added a comment.Nov 4 2025, 10:37 PM

Change #1201822 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: use notify to ensure service gets restarted on config changes

https://gerrit.wikimedia.org/r/1201822

gerritbot added a comment.Nov 4 2025, 11:05 PM

Change #1201820 merged by Dzahn:

[operations/puppet@production] gerrit: allow production networks to connect to gerrit-ssh

https://gerrit.wikimedia.org/r/1201820

gerritbot added a comment.Nov 4 2025, 11:08 PM

Change #1201822 merged by Dzahn:

[operations/puppet@production] tcpproxy: use notify to ensure service gets restarted on config changes

https://gerrit.wikimedia.org/r/1201822

gerritbot added a comment.Nov 4 2025, 11:21 PM

Change #1201828 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: add simple puppet service resource to manage haproxy

https://gerrit.wikimedia.org/r/1201828

gerritbot added a comment.Nov 4 2025, 11:38 PM

Change #1201828 merged by Dzahn:

[operations/puppet@production] tcpproxy: add simple puppet service resource to manage haproxy

https://gerrit.wikimedia.org/r/1201828

Dzahn added a comment.Nov 4 2025, 11:56 PM

haproxy configured as TCP-proxy for gerrit-ssh has been deployed on all 14 VMs, across POPs.

We can now connect to localhost 29418 and get gerrit-ssh on each of them.

[tcp-proxy5001:~] $ nc localhost 29418
SSH-2.0-GerritCodeReview_3.10.6 (APACHE-SSHD-2.12.0)

[tcp-proxy2001:~] $ nc localhost 29418
SSH-2.0-GerritCodeReview_3.10.6 (APACHE-SSHD-2.12.0)

..
Dzahn updated the task description. (Show Details)Nov 4 2025, 11:57 PM
gerritbot added a comment.Nov 5 2025, 2:28 PM

Change #1202152 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] tcpproxy: haproxy: make stats work

https://gerrit.wikimedia.org/r/1202152

gerritbot added a comment.Nov 5 2025, 2:31 PM

Change #1202152 merged by CDanis:

[operations/puppet@production] tcpproxy: haproxy: make stats work

https://gerrit.wikimedia.org/r/1202152

gerritbot added a comment.Nov 5 2025, 3:01 PM

Change #1202163 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] tcpproxy: haproxy: listen on v4+v6 for both ports

https://gerrit.wikimedia.org/r/1202163

gerritbot added a comment.Nov 5 2025, 3:19 PM

Change #1202172 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] tcpproxy: haproxy: log level change to info

https://gerrit.wikimedia.org/r/1202172

gerritbot added a comment.Nov 5 2025, 5:20 PM

Change #1202163 merged by Dzahn:

[operations/puppet@production] tcpproxy: haproxy: listen on v4+v6 for both ports

https://gerrit.wikimedia.org/r/1202163

gerritbot added a comment.Nov 5 2025, 5:22 PM

Change #1202172 merged by Dzahn:

[operations/puppet@production] tcpproxy: haproxy: log level change to info

https://gerrit.wikimedia.org/r/1202172

gerritbot added a comment.Nov 5 2025, 7:52 PM

Change #1202261 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: allow PRODUCTION_NETWORKS to connect to 29418

https://gerrit.wikimedia.org/r/1202261

gerritbot added a comment.Nov 5 2025, 7:55 PM

Change #1202261 merged by Dzahn:

[operations/puppet@production] tcpproxy: allow PRODUCTION_NETWORKS to connect to 29418

https://gerrit.wikimedia.org/r/1202261

Dzahn added a comment.EditedNov 5 2025, 9:25 PM

We are debugging why things (nc tcp-proxy* 29418) work from SOME of the VMs but not from others..in this pattern:

P84954 nc tcp-proxyXXXX.XXXX.wmnet 29418
1source | dest | works?
21001 | 1001 | YES
31001 | 1002 | NO
41001 | 2001 | YES
51001 | 2002 | YES
61001 | 3001 | NO
71001 | 3002 | NO
81001 | 4001 | YES
91001 | 4002 | YES
101001 | 5001 | YES
111001 | 5002 | YES
121001 | 6001 | YES
131001 | 6002 | YES
141001 | 7001 | NO
151001 | 7002 | NO
16
17
183001 | 1001 | NO
193001 | 1002 | NO
203001 | 2002 | YES
213001 | 2002 | YES
223001 | 3001 | NO
233001 | 3002 | NO
243001 | 4001 | YES
253001 | 4002 | YES
263001 | 5001 | YES
273001 | 5002 | YES
283001 | 6001 | YES
293001 | 6002 | YES
303001 | 7001 | NO
313001 | 7002 | NO
32
337001 | 1001 | NO
347001 | 1002 | NO
357001 | 2002 | YES
367001 | 2002 | YES
377001 | 3001 | NO
387001 | 3002 | NO
397001 | 4001 | YES
407001 | 4002 | YES
417001 | 5001 | YES
427001 | 5002 | YES
437001 | 6001 | YES
447001 | 6002 | YES
457001 | 7001 | NO
467001 | 7002 | NO

Dzahn reopened subtask T408064: Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy) as Open.Nov 5 2025, 11:24 PM
Dzahn added a comment.Nov 6 2025, 2:12 AM

We had some strange results when trying to debug this together. So I ended up testing every combination between the proxy VMs.

Using nc -z -w3 in a loop produces this type of output below.

Here it becomes obvious that esams and magru are IPv4-only and there are some other outliers. But connections do succed one way or another for all of them when testing like this.

P84963 tcp-proxies - verify every connection
11001
2
3root@tcp-proxy1001:/# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
4 nc -z -w3 tcp-proxy"${host}".wmnet 29418
5done
6Connection to tcp-proxy1001.eqiad.wmnet (10.64.32.131) 29418 port [tcp/*] succeeded!
7Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
8Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
9Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
10Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
11Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
12Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
13Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
14Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
15Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
16Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
17Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
18Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
19
20----
21
221002
23
24root@tcp-proxy1002:/# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
25 nc -z -w3 tcp-proxy"${host}".wmnet 29418
26done
27Connection to tcp-proxy1002.eqiad.wmnet (10.64.48.45) 29418 port [tcp/*] succeeded!
28Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
29Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
30Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
31Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
32Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
33Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
34Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
35Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
36Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
37Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
38Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
39Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
40
41----
42
432001
44
45root@tcp-proxy2001:/# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
46 nc -z -w3 tcp-proxy"${host}".wmnet 29418
47done
48Connection to tcp-proxy2001.codfw.wmnet (10.192.32.20) 29418 port [tcp/*] succeeded!
49Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
50Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
51Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
52Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
53Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
54Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
55Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
56Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
57Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
58Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
59Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
60
61----
62
632002
64
65root@tcp-proxy2002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
66 nc -z -w3 tcp-proxy"${host}".wmnet 29418
67done
68Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
69Connection to tcp-proxy2002.codfw.wmnet (10.192.48.15) 29418 port [tcp/*] succeeded!
70Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
71Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
72Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
73Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
74Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
75Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
76Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
77Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
78Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
79Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
80
81----
82
833001
84
85root@tcp-proxy3001:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
86 nc -z -w3 tcp-proxy"${host}".wmnet 29418
87done
88Connection to tcp-proxy2001.codfw.wmnet (10.192.32.20) 29418 port [tcp/*] succeeded!
89Connection to tcp-proxy2002.codfw.wmnet (10.192.48.15) 29418 port [tcp/*] succeeded!
90Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
91Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
92Connection to tcp-proxy4001.ulsfo.wmnet (10.128.0.17) 29418 port [tcp/*] succeeded!
93Connection to tcp-proxy4002.ulsfo.wmnet (10.128.0.23) 29418 port [tcp/*] succeeded!
94Connection to tcp-proxy5001.eqsin.wmnet (10.132.0.13) 29418 port [tcp/*] succeeded!
95Connection to tcp-proxy5002.eqsin.wmnet (10.132.0.21) 29418 port [tcp/*] succeeded!
96Connection to tcp-proxy6001.drmrs.wmnet (10.136.0.19) 29418 port [tcp/*] succeeded!
97Connection to tcp-proxy6002.drmrs.wmnet (10.136.1.18) 29418 port [tcp/*] succeeded!
98Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
99Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
100
101----
102
1033002
104
105root@tcp-proxy3002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
106 nc -z -w3 tcp-proxy"${host}".wmnet 29418
107done
108Connection to tcp-proxy2001.codfw.wmnet (10.192.32.20) 29418 port [tcp/*] succeeded!
109Connection to tcp-proxy2002.codfw.wmnet (10.192.48.15) 29418 port [tcp/*] succeeded!
110Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
111Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
112Connection to tcp-proxy4001.ulsfo.wmnet (10.128.0.17) 29418 port [tcp/*] succeeded!
113Connection to tcp-proxy4002.ulsfo.wmnet (10.128.0.23) 29418 port [tcp/*] succeeded!
114Connection to tcp-proxy5001.eqsin.wmnet (10.132.0.13) 29418 port [tcp/*] succeeded!
115Connection to tcp-proxy5002.eqsin.wmnet (10.132.0.21) 29418 port [tcp/*] succeeded!
116Connection to tcp-proxy6001.drmrs.wmnet (10.136.0.19) 29418 port [tcp/*] succeeded!
117Connection to tcp-proxy6002.drmrs.wmnet (10.136.1.18) 29418 port [tcp/*] succeeded!
118Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
119Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
120
121----
122
1234001
124
125root@tcp-proxy4001:/home/dzahn# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
126 nc -z -w3 tcp-proxy"${host}".wmnet 29418
127done
128
129Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
130Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
131Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
132Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
133Connection to tcp-proxy4001.ulsfo.wmnet (10.128.0.17) 29418 port [tcp/*] succeeded!
134Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
135Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
136Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
137Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
138Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
139Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
140Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
141
142----
143
1444002
145
146root@tcp-proxy4002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
147 nc -z -w3 tcp-proxy"${host}".wmnet 29418
148done
149Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
150Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
151Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
152Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
153Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
154Connection to tcp-proxy4002.ulsfo.wmnet (10.128.0.23) 29418 port [tcp/*] succeeded!
155Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
156Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
157Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
158Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
159Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
160Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
161
162----
163
1645001
165
166root@tcp-proxy5001:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
167 nc -z -w3 tcp-proxy"${host}".wmnet 29418
168done
169
170Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
171Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
172Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
173Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
174Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
175Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
176Connection to tcp-proxy5001.eqsin.wmnet (10.132.0.13) 29418 port [tcp/*] succeeded!
177Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
178Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
179Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
180Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
181Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
182
183----
184
1855002
186
187root@tcp-proxy5002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
188 nc -z -w3 tcp-proxy"${host}".wmnet 29418
189done
190
191Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
192Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
193Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
194Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
195Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
196Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
197Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
198Connection to tcp-proxy5002.eqsin.wmnet (10.132.0.21) 29418 port [tcp/*] succeeded!
199Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
200Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
201Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
202Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
203
204----
205
2066001
207
208root@tcp-proxy6001:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
209 nc -z -w3 tcp-proxy"${host}".wmnet 29418
210done
211
212Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
213Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
214Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
215Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
216Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
217Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
218Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
219Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
220Connection to tcp-proxy6001.drmrs.wmnet (10.136.0.19) 29418 port [tcp/*] succeeded!
221Connection to tcp-proxy6002.drmrs.wmnet (2a02:ec80:600:102:10:136:1:18) 29418 port [tcp/*] succeeded!
222Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
223Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
224
225----
226
2276002
228
229root@tcp-proxy6002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
230 nc -z -w3 tcp-proxy"${host}".wmnet 29418
231done
232
233Connection to tcp-proxy2001.codfw.wmnet (2620:0:860:103:10:192:32:20) 29418 port [tcp/*] succeeded!
234Connection to tcp-proxy2002.codfw.wmnet (2620:0:860:104:10:192:48:15) 29418 port [tcp/*] succeeded!
235Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
236Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
237Connection to tcp-proxy4001.ulsfo.wmnet (2620:0:863:101:10:128:0:17) 29418 port [tcp/*] succeeded!
238Connection to tcp-proxy4002.ulsfo.wmnet (2620:0:863:101:10:128:0:23) 29418 port [tcp/*] succeeded!
239Connection to tcp-proxy5001.eqsin.wmnet (2001:df2:e500:101:10:132:0:13) 29418 port [tcp/*] succeeded!
240Connection to tcp-proxy5002.eqsin.wmnet (2001:df2:e500:101:10:132:0:21) 29418 port [tcp/*] succeeded!
241Connection to tcp-proxy6001.drmrs.wmnet (2a02:ec80:600:101:10:136:0:19) 29418 port [tcp/*] succeeded!
242Connection to tcp-proxy6002.drmrs.wmnet (10.136.1.18) 29418 port [tcp/*] succeeded!
243Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
244Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
245
246----
247
2487001
249
250root@tcp-proxy7001:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
251 nc -z -w3 tcp-proxy"${host}".wmnet 29418
252done
253Connection to tcp-proxy2001.codfw.wmnet (10.192.32.20) 29418 port [tcp/*] succeeded!
254Connection to tcp-proxy2002.codfw.wmnet (10.192.48.15) 29418 port [tcp/*] succeeded!
255Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
256Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
257Connection to tcp-proxy4001.ulsfo.wmnet (10.128.0.17) 29418 port [tcp/*] succeeded!
258Connection to tcp-proxy4002.ulsfo.wmnet (10.128.0.23) 29418 port [tcp/*] succeeded!
259Connection to tcp-proxy5001.eqsin.wmnet (10.132.0.13) 29418 port [tcp/*] succeeded!
260Connection to tcp-proxy5002.eqsin.wmnet (10.132.0.21) 29418 port [tcp/*] succeeded!
261Connection to tcp-proxy6001.drmrs.wmnet (10.136.0.19) 29418 port [tcp/*] succeeded!
262Connection to tcp-proxy6002.drmrs.wmnet (10.136.1.18) 29418 port [tcp/*] succeeded!
263Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
264Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!
265
266----
267
2687002
269
270root@tcp-proxy7002:~# for host in 1001.eqiad 1002.eqiad 2001.codfw 2002.codfw 3001.esams 3002.esams 4001.ulsfo 4002.ulsfo 5001.eqsin 5002.eqsin 6001.drmrs 6002.drmrs 7001.magru 7002.magru; do
271 nc -z -w3 tcp-proxy"${host}".wmnet 29418
272done
273
274Connection to tcp-proxy2001.codfw.wmnet (10.192.32.20) 29418 port [tcp/*] succeeded!
275Connection to tcp-proxy2002.codfw.wmnet (10.192.48.15) 29418 port [tcp/*] succeeded!
276Connection to tcp-proxy3001.esams.wmnet (10.80.2.9) 29418 port [tcp/*] succeeded!
277Connection to tcp-proxy3002.esams.wmnet (10.80.2.10) 29418 port [tcp/*] succeeded!
278Connection to tcp-proxy4001.ulsfo.wmnet (10.128.0.17) 29418 port [tcp/*] succeeded!
279Connection to tcp-proxy4002.ulsfo.wmnet (10.128.0.23) 29418 port [tcp/*] succeeded!
280Connection to tcp-proxy5001.eqsin.wmnet (10.132.0.13) 29418 port [tcp/*] succeeded!
281Connection to tcp-proxy5002.eqsin.wmnet (10.132.0.21) 29418 port [tcp/*] succeeded!
282Connection to tcp-proxy6001.drmrs.wmnet (10.136.0.19) 29418 port [tcp/*] succeeded!
283Connection to tcp-proxy6002.drmrs.wmnet (10.136.1.18) 29418 port [tcp/*] succeeded!
284Connection to tcp-proxy7001.magru.wmnet (10.140.2.10) 29418 port [tcp/*] succeeded!
285Connection to tcp-proxy7002.magru.wmnet (10.140.2.11) 29418 port [tcp/*] succeeded!

Dzahn mentioned this in T408064: Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy).Nov 6 2025, 2:35 AM
hashar unsubscribed.Nov 6 2025, 9:11 AM
gerritbot added a comment.Nov 6 2025, 7:32 PM

Change #1202782 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] allocate eqiad VIP for load balanced tcp-proxy service

https://gerrit.wikimedia.org/r/1202782

gerritbot added a comment.Nov 6 2025, 9:03 PM

Change #1202782 merged by Dzahn:

[operations/dns@master] allocate eqiad VIP for load balanced tcp-proxy service

https://gerrit.wikimedia.org/r/1202782

gerritbot added a comment.Nov 6 2025, 9:34 PM

Change #1202835 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] allocate codfw VIP for load-balanced tcp-proxy service

https://gerrit.wikimedia.org/r/1202835

gerritbot added a comment.Nov 6 2025, 9:48 PM

Change #1202842 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] service: add tcpproxy service to service catalog (WIP)

https://gerrit.wikimedia.org/r/1202842

gerritbot added a comment.Nov 7 2025, 3:57 PM

Change #1202835 merged by Dzahn:

[operations/dns@master] allocate codfw VIP for load-balanced tcp-proxy service

https://gerrit.wikimedia.org/r/1202835

CDanis closed subtask T408064: Site: 14 VMs request for tcp-proxy (gerrit-ssh-proxy) as Resolved.Nov 7 2025, 5:59 PM
gerritbot added a comment.Nov 7 2025, 10:29 PM

Change #1203157 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: include profile::lvs::realserver in role

https://gerrit.wikimedia.org/r/1203157

Dzahn added a comment.Nov 14 2025, 5:04 PM

A proxy is running on 14 VMs, 2 in each of the 7 POPs.

What is missing is the load-balancing part.

LSobanski moved this task from Work in Progress (Tracking tasks) to Work in Progress on the collaboration-services board.Nov 20 2025, 12:11 PM
gerritbot added a comment.Dec 2 2025, 1:24 PM

Change #1198281 abandoned by Jelto:

[operations/puppet@production] git_ssh_proxy: add role::git_ssh_proxy for Gerrit and GitLab ssh proxies

Reason:

not needed anymore in favor of haproxy

https://gerrit.wikimedia.org/r/1198281

gerritbot added a comment.Dec 4 2025, 7:09 PM

Change #1202842 merged by Dzahn:

[operations/puppet@production] service: add gerrit-https service to service catalog

https://gerrit.wikimedia.org/r/1202842

gerritbot added a comment.Dec 4 2025, 7:45 PM

Change #1203157 abandoned by Dzahn:

[operations/puppet@production] tcpproxy: include profile::lvs::realserver in role

Reason:

replaced by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1215240

https://gerrit.wikimedia.org/r/1203157

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2025, 8:32 PM
Dzahn added a comment.Dec 4 2025, 10:12 PM

We have to switch these hosts from nftables back to ferm as firewall provider. Reason: liberica does not support nftables yet.

gerritbot added a comment.Dec 4 2025, 10:16 PM

Change #1215284 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] tcpproxy: switch firewall provider from nftables to ferm

https://gerrit.wikimedia.org/r/1215284

gerritbot added a project: Patch-For-Review.Dec 4 2025, 10:16 PM
Stashbot added a comment.Dec 4 2025, 10:22 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-04T22:22:16Z] <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: T408532

gerritbot added a comment.Dec 4 2025, 10:23 PM

Change #1215284 merged by Dzahn:

[operations/puppet@production] tcpproxy: switch firewall provider from nftables to ferm

https://gerrit.wikimedia.org/r/1215284

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2025, 10:30 PM
Dzahn added a comment.Dec 4 2025, 11:05 PM

downtimed, ran puppet, rebooted the 14 VMs and verified ferm service is running via cumin/cookbook. they are all on ferm now.

Dzahn added a comment.Dec 5 2025, 12:03 AM

This change should have been linked here. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1215240 (thanks cdanis!)

It added the LVS profiles to the tcp-proxy puppet role.

The 14 VMs now have the new gerrit-lb IP, v4 and v6, bound on their loopback interfaces, 7 pairs:

[cumin2002:~] $ sudo cumin 'tcp-*' "ip addr show dev lo | grep global"
14 hosts will be targeted:
tcp-proxy[2001-2002].codfw.wmnet,tcp-proxy[6001-6002].drmrs.wmnet,tcp-proxy[1001-1002].eqiad.wmnet,tcp-proxy[5001-5002].eqsin.wmnet,tcp-proxy[3001-3002].esams.wmnet,tcp-proxy[7001-7002].magru.wmnet,tcp-proxy[4001-4002].ulsfo.wmnet
OK to proceed on 14 hosts? Enter the number of affected hosts to confirm or "q" to quit: 14
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[5001-5002].eqsin.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 103.102.166.225/32 scope global lo:LVS                                                                                                                                               
    inet6 2001:df2:e500:ed1a::2/128 scope global                                                                                                                                              
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[7001-7002].magru.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 195.200.68.225/32 scope global lo:LVS                                                                                                                                                
    inet6 2a02:ec80:700:ed1a::2/128 scope global                                                                                                                                              
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[6001-6002].drmrs.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 185.15.58.225/32 scope global lo:LVS                                                                                                                                                 
    inet6 2a02:ec80:600:ed1a::2/128 scope global                                                                                                                                              
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[3001-3002].esams.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 185.15.59.225/32 scope global lo:LVS                                                                                                                                                 
    inet6 2a02:ec80:300:ed1a::2/128 scope global                                                                                                                                              
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[4001-4002].ulsfo.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 198.35.26.97/32 scope global lo:LVS                                                                                                                                                  
    inet6 2620:0:863:ed1a::2/128 scope global                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[1001-1002].eqiad.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 208.80.154.225/32 scope global lo:LVS                                                                                                                                                
    inet6 2620:0:861:ed1a::2/128 scope global                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                        
(2) tcp-proxy[2001-2002].codfw.wmnet                                                                                                                                                          
----- OUTPUT of 'ip addr show dev lo | grep global' -----                                                                                                                                     
    inet 208.80.153.225/32 scope global lo:LVS                                                                                                                                                
    inet6 2620:0:860:ed1a::2/128 scope global
Dzahn added a comment.Dec 5 2025, 12:05 AM

This should conclude the box:

Prepare tcpproxy VMs for accepting traffic on the new public IPs

on the parent task "Move Gerrit behind the CDN".

And this ticket should be resolved.

Dzahn closed this task as Resolved.Dec 5 2025, 12:06 AM

from here on anything would be just updating 2 tickets at a time. This is done and if there are small follow-ups they might happen over in T365259

gerritbot added a comment.Jan 7 2026, 11:59 AM

Change #1224057 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add Cumin alias for tcpproxy hosts

https://gerrit.wikimedia.org/r/1224057

gerritbot added a project: Patch-For-Review.Jan 7 2026, 11:59 AM
gerritbot added a comment.Jan 8 2026, 7:44 AM

Change #1224057 merged by Muehlenhoff:

[operations/puppet@production] Add Cumin alias for tcpproxy hosts

https://gerrit.wikimedia.org/r/1224057

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2026, 8:38 AM
gerritbot added a comment.Jan 19 2026, 2:34 PM

Change #1228515 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: change healthcheck URL for service catalog

https://gerrit.wikimedia.org/r/1228515

gerritbot added a project: Patch-For-Review.Jan 19 2026, 2:34 PM
gerritbot added a comment.Jan 19 2026, 2:43 PM

Change #1228515 merged by Arnaudb:

[operations/puppet@production] gerrit: change healthcheck URL for service catalog

https://gerrit.wikimedia.org/r/1228515

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2026, 3:31 PM
Dzahn added a parent task: T411895: gerrit behind CDN.Feb 9 2026, 8:17 PM
hashar mentioned this in T417497: Gerrit events not received by Zuul due to TCP Proxy timeout (CI is not triggered for some patches).Feb 17 2026, 2:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits