Whenever upgrading the zuul debian package on the spare CI server (contint2001), the zuul server is started and establishes connections to Gerrit. That in turns fill in two SSH connections on Gerrit but since we limit the pool to 4 connections per server, the zuul-merger can no more fetches from Gerrit. That caused T233390.
I have mentioned the issue on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/502203/ :
The reason for the change is that each time I upgrade the Zuul debian package, the service is started despite having:
hieradata/hosts/contint1001.yaml:profile::zuul::server::service_ensure: running hieradata/hosts/contint2001.yaml:profile::zuul::server::service_ensure: stopped
And indeed the next puppet run would stop it:
Notice: /Stage[main]/Zuul::Server/Systemd::Service[zuul]/Service[zuul]/ensure: ensure changed 'running' to 'stopped'
But the package installation starts it nonetheless:
root@contint2001:~# dpkg -i zuul_2.5.1-wmf9_amd64.deb ... Processing triggers for systemd (215-17+deb8u13) ... root@contint2001:~# systemctl status zuul Active: active (running) since Fri 2019-05-17 08:32:23 UTC; 4s ago
I guess I would need it to be masked instead?