Ok, will do!

Change #1305488 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: configure upstream_host: "localhost" for envoy

https://gerrit.wikimedia.org/r/1305488

@hashar regarding the test with netcat:

This is actually an IPv4 vs IPv6 issue:

nc -v -l -p 8080 will only listen on IPv4. (Listening on 0.0.0.0 8080).

[contint1003:~] $ nc -zv 2620:0:861:2:208:80:154:137 8080
nc: connect to 2620:0:861:2:208:80:154:137 port 8080 (tcp) failed: Connection refused

[contint1003:~] $ nc -6zv 2620:0:861:2:208:80:154:137 8080
nc: connect to 2620:0:861:2:208:80:154:137 port 8080 (tcp) failed: Connection refused

[contint1003:~] $ nc -4zv 208.80.154.137 8080
Connection to 208.80.154.137 8080 port [tcp/http-alt] succeeded!

envoy with strict_dns will resolve contint1003.wikimedia.org and prefer IPv6.

and this works:

nc -6 -v -l -p 8080
Listening on :: 8080

[contint1003:~] $ nc -zv 2620:0:861:2:208:80:154:137 8080
Connection to 2620:0:861:2:208:80:154:137 8080 port [tcp/http-alt] succeeded!

[contint1003:~] $ host 2620:0:861:2:208:80:154:137
7.3.1.0.4.5.1.0.0.8.0.0.8.0.2.0.2.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer contint1003.wikimedia.org.

So I don't think that was the right test.

regardless; let me configure the upstream_host to use localhost instead:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305488

https://puppet-compiler.wmflabs.org/output/1305488/8778/contint1003.wikimedia.org/index.html

Envoyproxy::Cluster[cluster_local_port_8080]

Envoyproxy::Tls_terminator[1443]

    Parameters differences:

    --- Envoyproxy::Tls_terminator[1443].orig
    +++ Envoyproxy::Tls_terminator[1443]

    @@
    -    upstreams => [{'server_names': ['*'], 'cert_path': None, 'key_path': None, 'upstream_port': 8080, 'upstream_addr': 'contint1003.wikimedia.org'}]
    +    upstreams => [{'server_names': ['*'], 'cert_path': None, 'key_path': None, 'upstream_port': 8080, 'upstream_addr': 'localhost'}]

Envoyproxy::Conf[cluster_local_port_8080]

File[/etc/envoy/clusters.d/00-cluster_local_port_8080.yaml]

    Content differences:

    --- /etc/envoy/clusters.d/00-cluster_local_port_8080.yaml.orig
    +++ /etc/envoy/clusters.d/00-cluster_local_port_8080.yaml
    @@ -9,5 +9,5 @@
         - endpoint:
             address:
               socket_address:
    -            address: contint1003.wikimedia.org
    +            address: localhost
                 port_value: 8080

Class[Profile::Tlsproxy::Envoy]

    Parameters differences:

    --- Class[Profile::Tlsproxy::Envoy].orig
    +++ Class[Profile::Tlsproxy::Envoy]

    @@
    -    upstream_addr => contint1003.wikimedia.org
    +    upstream_addr => localhost

Change #1305488 merged by Dzahn:

[operations/puppet@production] jenkins: configure upstream_addr as localhost for envoy

https://gerrit.wikimedia.org/r/1305488

@hashar The change above has been made. Can we start jenkins to test again and with the real backend service now?

Change #1300916 merged by Dzahn:

[operations/puppet@production] contint: add second proxy for jenkins on an external host

https://gerrit.wikimedia.org/r/1300916

Change #1305515 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add a missing trailing slash to close the Proxy block

https://gerrit.wikimedia.org/r/1305515

Change #1305515 merged by Dzahn:

[operations/puppet@production] jenkins: add a missing trailing slash to close the Proxy block

https://gerrit.wikimedia.org/r/1305515

Change #1305531 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: load mod_ssl in httpd to be able to proxy https

https://gerrit.wikimedia.org/r/1305531

With gerrit:1300916 I have introduced a new config file /etc/apache2/jenkins_proxy_ext on contint1002/2002.

The intention of this file is to proxy the path /jenkins to our new external jenkins just like the existing path /ci proxies to the existing jenkins on localhost.

And the further goal of that is that we are able to test the connection to the new jenkins as long as we want and it does not have to happen during a maintenance window where we take regular CI down.

This is not working yet and we need to load the SSL module and activate it and probably make other changes but there is no reason for concern, since this file is currently not included in the apache config.

This is because in sites-enabled/50-integration-wikimedia-org.conf we Include *_proxy but this does NOT match jenkins_proxy_ext.

It would only be included by changing that Include line to, for example, Include *_proxy*.

I will keep working on this but make any tests on the secondary server, so contint2002 and not contint1002.

In T418521#12051810, @Dzahn wrote:

@hashar regarding the test with netcat:

This is actually an IPv4 vs IPv6 issue:

nc -v -l -p 8080 will only listen on IPv4. (Listening on 0.0.0.0 8080).

[contint1003:~] $ nc -zv 2620:0:861:2:208:80:154:137 8080
nc: connect to 2620:0:861:2:208:80:154:137 port 8080 (tcp) failed: Connection refused

[contint1003:~] $ nc -6zv 2620:0:861:2:208:80:154:137 8080
nc: connect to 2620:0:861:2:208:80:154:137 port 8080 (tcp) failed: Connection refused

[contint1003:~] $ nc -4zv 208.80.154.137 8080
Connection to 208.80.154.137 8080 port [tcp/http-alt] succeeded!

envoy with strict_dns will resolve contint1003.wikimedia.org and prefer IPv6.

and this works:

nc -6 -v -l -p 8080
Listening on :: 8080

[contint1003:~] $ nc -zv 2620:0:861:2:208:80:154:137 8080
Connection to 2620:0:861:2:208:80:154:137 8080 port [tcp/http-alt] succeeded!

[contint1003:~] $ host 2620:0:861:2:208:80:154:137
7.3.1.0.4.5.1.0.0.8.0.0.8.0.2.0.2.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer contint1003.wikimedia.org.

So I don't think that was the right test.

Awesome what a great catch! thank you!

Can we start Jenkins to test again and with the real backend service now?

The Jenkins config has been synced to contint1003 it thus has the jobs that run on a timer which would start failing. That is why I went with nc to simulate the connection.

With the Envoy config fix when I do:

curl -v --connect-to ::contint1003.wikimedia.org   https://jenkins.discovery.wmnet:1443/

On the other side:

$ nc -v -l -46 -p 8080
Listening on :: 8080
Connection received on localhost 54426
GET / HTTP/1.1
host: jenkins.discovery.wmnet:1443
user-agent: curl/7.74.0
accept: */*
x-forwarded-proto: https
x-request-id: e680c927-1bcb-4627-8250-1ca9abd9a79d
x-envoy-expected-rq-timeout-ms: 65000

Success

In T418521#12052337, @Dzahn wrote:

With gerrit:1300916 I have introduced a new config file /etc/apache2/jenkins_proxy_ext on contint1002/2002.

The intention of this file is to proxy the path /jenkins to our new external jenkins just like the existing path /ci proxies to the existing jenkins on localhost.

And the further goal of that is that we are able to test the connection to the new jenkins as long as we want and it does not have to happen during a maintenance window where we take regular CI down.

This is not working yet and we need to load the SSL module and activate it and probably make other changes but there is no reason for concern, since this file is currently not included in the apache config.

This is because in sites-enabled/50-integration-wikimedia-org.conf we Include *_proxy but this does NOT match jenkins_proxy_ext.

It would only be included by changing that Include line to, for example, Include *_proxy*.

To start the Jenkins instance on contint1003, I think the easiest is to delete the jobs and nodes configurations, this way we don't have any job running on the new instance. That is deleting: /var/lib/jenkins/jobs /var/lib/jenkins/nodes. Then start Jenkins.

Given that new Jenkins instance would not be doing much, I think it is fine to expose it to a new URL /jenkins by renaming the Apache config snippet from jenkins_proxy_ext to proxy_jenkins_ext. The web UI might show defect cause the Jenkins configuration refers to /ci which might break some links, but at least the default landing page should show something which is enough to validate Envoy (done already) and Apache config.

From there we can rescheduled a switch over next week?

Change #1305531 merged by Dzahn:

[operations/puppet@production] ci: load mod_ssl in httpd to be able to proxy https

https://gerrit.wikimedia.org/r/1305531

Change #1305759 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] integration: include all proxy configs, including new for jenkins

https://gerrit.wikimedia.org/r/1305759

Change #1305759 merged by Dzahn:

[operations/puppet@production] integration: include all proxy configs, including new for jenkins

https://gerrit.wikimedia.org/r/1305759

Change #1305767 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] integration: turn SSLProxyEngine on

https://gerrit.wikimedia.org/r/1305767

Change #1305767 merged by Dzahn:

[operations/puppet@production] integration: turn SSLProxyEngine on

https://gerrit.wikimedia.org/r/1305767

@hashar

• let apache httpd load all the proxy configs
• loaded ssl module
• turned SSLProxyEngine on
• restarted apache httpd

Now I can simulate being jenkins using nc like this:

printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 18\r\nConnection: close\r\n\r\nHello from netcat!" | nc -6 -l 8080

And it works. I can see:

note: this only works with nc -6 when using IPv6

Change #1305772 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: enable jenkins service

https://gerrit.wikimedia.org/r/1305772

Change #1305772 merged by Dzahn:

[operations/puppet@production] jenkins: enable jenkins service

https://gerrit.wikimedia.org/r/1305772

Setup/fixed the apache->envoy->jenkins proxy chain from contint1002->contint1003 over SSL.

Deleted the jenkins jobs directory so it does nothing and then activated the service again. (it is running on contint1003 and masked on contint2003).

The new jenkins is now available at:

https://integration.wikimedia.org/jenkins

(and old jenkins unchanged at https://integration.wikimedia.org/ci as before)

masked the service again because I was asked to deactivate it in case there is a side effect. but we now have confirmed it works.

Change #1306445 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: in httpd include proxy configs individually, not by wildcard

https://gerrit.wikimedia.org/r/1306445

Change #1306446 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: stop including local jenkins proxy, switch ext proxy to /ci

https://gerrit.wikimedia.org/r/1306446

Change #1306445 merged by Dzahn:

[operations/puppet@production] contint: in httpd include proxy configs individually, not by wildcard

https://gerrit.wikimedia.org/r/1306445

Mentioned in SAL (#wikimedia-operations) [2026-07-02T19:48:33Z] <mutante> contint2003 - reimaging because of T430510#12067628 T418521

Mentioned in SAL (#wikimedia-operations) [2026-07-02T22:03:33Z] <mutante> contint1003 (zuul.wikimedia.org) - reimaging because of T430510#12067628 T418521

Change #1307850 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: disable jenkins on role::ci machines

https://gerrit.wikimedia.org/r/1307850

added new maintenance window, July 7th, 15:30–16:30 UTC # 08:30–09:30 PDT to Deployment Calendar

https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_July_7

runbook v2 https://docs.google.com/document/d/1dR2k4Ea2p67Fj12iZALQh1BwlHe5bKfhp7esg7Tc2VM/edit?usp=sharing

Change #1307850 merged by Dzahn:

[operations/puppet@production] ci: disable jenkins on role::ci machines

https://gerrit.wikimedia.org/r/1307850

Change #1306446 merged by Dzahn:

[operations/puppet@production] contint: stop including local jenkins proxy, switch ext proxy to /ci

https://gerrit.wikimedia.org/r/1306446

Change #1308153 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: switch jenkins prefix to /ci on new hosts

https://gerrit.wikimedia.org/r/1308153

Change #1308153 merged by Dzahn:

[operations/puppet@production] jenkins: switch jenkins prefix to /ci on new hosts

https://gerrit.wikimedia.org/r/1308153

Change #1308154 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: enable jenkins service on new host

https://gerrit.wikimedia.org/r/1308154

Change #1308154 merged by Dzahn:

[operations/puppet@production] jenkins: enable jenkins service on new host

https://gerrit.wikimedia.org/r/1308154

Change #1308158 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add firewall rule for controller to agent

https://gerrit.wikimedia.org/r/1308158

Change #1308158 merged by Dzahn:

[operations/puppet@production] jenkins: add firewall rule for controller to agent

https://gerrit.wikimedia.org/r/1308158

Change #1308164 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: include profile for jenkins agent and user setup

https://gerrit.wikimedia.org/r/1308164

Change #1308164 merged by Dzahn:

[operations/puppet@production] jenkins: include profile for jenkins agent and user setup

https://gerrit.wikimedia.org/r/1308164

Change #1308165 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add Hiera keys for jenkins agent

https://gerrit.wikimedia.org/r/1308165

Change #1308165 merged by Dzahn:

[operations/puppet@production] jenkins: add Hiera keys for jenkins agent

https://gerrit.wikimedia.org/r/1308165

Change #1308166 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add contint1003 to jenkins_controller_hosts

https://gerrit.wikimedia.org/r/1308166

Change #1308166 merged by Dzahn:

[operations/puppet@production] jenkins: add contint1003 to jenkins_controller_hosts

https://gerrit.wikimedia.org/r/1308166

Change #1308169 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: ensure workspaces_dir exists

https://gerrit.wikimedia.org/r/1308169

Change #1308170 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: use literal qualified path for workspace dir

https://gerrit.wikimedia.org/r/1308170

Change #1308172 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Switch Jenkins from contint1002 to contint1003

https://gerrit.wikimedia.org/r/1308172

Change #1308170 merged by Dzahn:

[operations/puppet@production] jenkins: use literal qualified path for workspace dir

https://gerrit.wikimedia.org/r/1308170

Change #1308169 abandoned by Dzahn:

[operations/puppet@production] jenkins: ensure workspaces_dir exists

Reason:

not needed

https://gerrit.wikimedia.org/r/1308169

Change #1308172 merged by jenkins-bot:

[integration/config@master] Switch Jenkins from contint1002 to contint1003

https://gerrit.wikimedia.org/r/1308172

https://docs.google.com/document/d/e/2PACX-1vQUiV0n8D2BcUKEFe4I3sxQKoYCc6g5nzrthtt_U-m23QC2-0yZSzXWFSyKIjsSY7W_CKx6iV0E17Pr/pub

Very well done @Dzahn thank you so much for having prepared all that migration and for the quiet detailed runbook.

Change #1308186 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: add contint1003 to list of firewall_hosts allowed to git over 9418

https://gerrit.wikimedia.org/r/1308186

Change #1308186 merged by Dzahn:

[operations/puppet@production] ci: add contint1003 to list of firewall_hosts allowed to git over 9418

https://gerrit.wikimedia.org/r/1308186

Change #1308194 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci::docker: ensure jenkins agent user is in docker group also in prod

https://gerrit.wikimedia.org/r/1308194

Thank you @hashar for the nice words and the migration today! :)

There were some follow-up fixes you can see above.

I noticed them when Amir reported this failing job thumbor-plugins-pipeline-test/355 and the console showed unable to connect to contint2002.wikimedia.org and it means the actual git:// protocol over port 9418, not ssh or https.

Then found in profile::zuul::merger there is:

# We run a git-daemon process to expose the zuul-merger git repositories.
# The slaves fetch changes from it over the git:// protocol.
# It is only meant to be used from slaves, so only accept internal
# connections.
if $firewall_hosts {
    firewall::service { 'git_daemon_internal_hosts':
        proto  => 'tcp',
        port   => 9418,
        srange => $firewall_hosts,

So the fix was just a Hiera edit but should have maybe added contint2003 as well.

Then once the firewall was open there was the new error permission denied while trying to connect to the Docker daemon socket. as can be seen in build 358.

I realized that's because the $jenkins_agent_user wasn't in the docker group. To verify I manually added it to the group, on both servers, and that fixed the issue.

When starting to make the puppet change to fix it the right away I found we already have the code for that, but limited to cloud, for some reason.

So there is still gerrit:1308194 to be reviewed/merged.

Change #1308245 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: add contint2003 to list of hosts allowed to git/9148

https://gerrit.wikimedia.org/r/1308245

Change #1308249 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: add parameter to toggle jenkins package install, remove it from role::ci

https://gerrit.wikimedia.org/r/1308249

Change #1308194 merged by Dzahn:

[operations/puppet@production] ci::docker: ensure jenkins agent user is in docker group also in prod

https://gerrit.wikimedia.org/r/1308194

Change #1308245 merged by Dzahn:

[operations/puppet@production] ci: add contint2003 to list of hosts allowed to git/9148

https://gerrit.wikimedia.org/r/1308245

Change #1309135 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] docker_registry: allow push from new contint hosts

https://gerrit.wikimedia.org/r/1309135

Change #1309135 merged by Clément Goubert:

[operations/puppet@production] docker_registry: allow push from new contint hosts

https://gerrit.wikimedia.org/r/1309135

Change #1311477 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] jenkins: enable overlay/overlayfs kernel modules for Docker

https://gerrit.wikimedia.org/r/1311477

Change #1311477 merged by Dzahn:

[operations/puppet@production] jenkins: enable overlay/overlayfs kernel modules for Docker

https://gerrit.wikimedia.org/r/1311477

Change #1318370 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: remove jenkins profiles from role::ci

https://gerrit.wikimedia.org/r/1318370

Change #1319130 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: stop including the legacy jenkins proxy config

https://gerrit.wikimedia.org/r/1319130

Change #1319130 merged by Dzahn:

[operations/puppet@production] ci: stop including the legacy jenkins proxy config

https://gerrit.wikimedia.org/r/1319130

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:03:48Z] <mutante> CI: contint1002/contint2002 - rm /etc/apache2/jenkins_proxy - removing legacy jenkins proxy config - jenkins is on new dedicated machines and uses jenkins_proxy_ext config T418521

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:05:12Z] <mutante> CI: contint1002/contint2002 - restarted httpd to be extra sure all is cleaned up - https://integration.wikimedia.org/ci/ is up and running T418521

Change #1318370 merged by Dzahn:

[operations/puppet@production] ci: remove jenkins profile from role::ci

https://gerrit.wikimedia.org/r/1318370

Mentioned in SAL (#wikimedia-operations) [2026-07-29T18:15:02Z] <mutante> CI: contint1002/contint2002: apt-get remove --purge jenkins - jenkins be gone - T418521

To me this was truly the final step in resolving this ticket:

The jenkins package has been purged from contint1002/contint2002.

No more jenkins on bullseye.

Change #1308249 abandoned by Dzahn:

[operations/puppet@production] ci: add parameter to toggle jenkins package install, remove it from role::ci

Reason:

not needed anymore. the package was removed with APT with --purge. we stopped including the profiles installing it and that also removed some firewall rules and monitoring.

https://gerrit.wikimedia.org/r/1308249

Change #1319839 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: allow inbound ssh from controller on prod agents

https://gerrit.wikimedia.org/r/1319839

Change #1319839 merged by Dzahn:

[operations/puppet@production] ci: allow inbound ssh from controller on prod agents

https://gerrit.wikimedia.org/r/1319839

Change #1320916 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: allow inbound ssh from controller on prod agent [2]

https://gerrit.wikimedia.org/r/1320916

Mentioned in SAL (#wikimedia-releng) [2026-08-24T08:26:55Z] <hashar> integration: removing firewall rule for ssh from contint1002/contint2002 IPv4 # T418521

Mentioned in SAL (#wikimedia-operations) [2026-09-01T14:03:19Z] <hashar> Removed openjdk-17 packages from contint1002/contint2002 following relocation of CI Jenkins to contint1003/contint2003 # T418521