The 6 VMs created for new zuul in T393873 need further setup / puppetization.
To avoid amending to the VM request ticket forever but have something to link changes to, continue on this new ticket.
This ticket covers further setup for all 3 types of new zuul VMs, main, executor and trusted job runner.
all changes in topic branch: https://gerrit.wikimedia.org/r/q/topic:%22zuul-new%22
Change #1208441 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add $service_ensure parameter for zuul services (WIP)
Change #1208441 merged by Dzahn:
[operations/puppet@production] zuul: add $service_ensure parameter for zuul services (WIP)
Change #1224908 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: add ssl.keyStore.passwordPath
Change #1227735 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: write TLS passphrase to a file for zookeeper
Change #1227735 merged by Dzahn:
[operations/puppet@production] zuul: write TLS passphrase to a file for zookeeper
Change #1233697 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: add parameter and path to tls cert passphrase
Change #1233697 abandoned by Dzahn:
[operations/puppet@production] zookeeper: add parameter and path to tls cert passphrase
Reason:
merge into https://gerrit.wikimedia.org/r/c/operations/puppet/+/1224908
Change #1224908 merged by Dzahn:
[operations/puppet@production] zookeeper: add ssl.keyStore.passwordPath
Change #1236730 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper/zuul: use standard port 2281 for TLS secureClientPort
Change #1236735 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: set keystore format to PKCS12 when enabling TLS (for zuul)
Change #1236730 merged by Dzahn:
[operations/puppet@production] zookeeper/zuul: use standard port 2281 for TLS secureClientPort
Change #1236735 merged by Dzahn:
[operations/puppet@production] zookeeper: set keystore format to PKCS12 when enabling TLS (for zuul)
Change #1236809 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: move .p12 keystore file under the zookeeper config path
Change #1236809 merged by Dzahn:
[operations/puppet@production] zuul: move .p12 keystore file under the zookeeper config path
Change #1236815 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use chained certificate incl CA for zookeeper
Change #1236815 merged by Dzahn:
[operations/puppet@production] zuul: use chained certificate incl CA for zookeeper
Change #1237253 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: mTLS debugging, use TLSv.1.2, clientAuth=want, set alias to 1
Change #1237253 merged by Dzahn:
[operations/puppet@production] zookeeper: mTLS debugging, use TLSv.1.2, clientAuth=want, set alias to 1
Change #1237294 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: create shared config dir for zookeeper-zuul mTLS
Change #1237294 merged by Dzahn:
[operations/puppet@production] zuul: create shared config dir for zookeeper-zuul mTLS
Change #1237327 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: parameterize and configure new config dir for zookeeper-zuul
Change #1237327 merged by Dzahn:
[operations/puppet@production] zuul: parameterize and configure new config dir for zookeeper-zuul
Change #1237341 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul/zookeeper: use CA-only, not chained file, as truststore
Change #1237341 merged by Dzahn:
[operations/puppet@production] zuul/zookeeper: use CA-only, not chained file, as truststore
Change #1237342 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper/zuul: use ssl.trustStore.password instead ssl.trustStore.passwordPath
Change #1237343 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/private@master] zookeeper: add fake TLS password to match private repo
Change #1237343 merged by Dzahn:
[labs/private@master] zookeeper: add fake TLS password to match private repo
Change #1237345 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/private@master] zuul: fix renamed password variable
Change #1237345 merged by Dzahn:
[labs/private@master] zuul: fix renamed password variable
Change #1237342 merged by Dzahn:
[operations/puppet@production] zookeeper/zuul: use ssl.trustStore.password instead ssl.trustStore.passwordPath
Change #1237354 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul-web: bind mount /etc/zookeeper/zuul-tls
Change #1237354 merged by Dzahn:
[operations/puppet@production] zuul-web: bind mount /etc/zookeeper/zuul-tls
Change #1237543 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: properly differentiate between zuul and zookeeper certs
Change #1237543 merged by Dzahn:
[operations/puppet@production] zuul: properly differentiate between zuul and zookeeper certs
Change #1237983 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: add Hiera key for tls_config_dir
Change #1237983 merged by Dzahn:
[operations/puppet@production] zuul::executor: add Hiera key for tls_config_dir
Change #1243984 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: do not use /etc/zookeeper as cert dir
Change #1243984 merged by Dzahn:
[operations/puppet@production] zuul::executor: do not use /etc/zookeeper as cert dir
Change #1244021 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: set tls_truststore for zookeeper to the copy it owns
Change #1244021 merged by Dzahn:
[operations/puppet@production] zuul::main: set tls_truststore for zookeeper to the copy it owns
Change #1244033 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: add extra Java opts to debug zookeeper TLS
Change #1244927 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: support TLS by loading Netty jars into class path
Change #1244939 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper::server: allow Hiera to override $extra_java_opts
Change #1244944 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: add debugging extra_java_opts: "-Djavax.net.debug=ssl,handshake"
Change #1244969 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: build full chain of trust for Java Netty TLS
Change #1244969 merged by Dzahn:
[operations/puppet@production] zuul::main: build full chain of trust for Java Netty TLS
Change #1244939 merged by Dzahn:
[operations/puppet@production] zookeeper::server: allow Hiera to override $extra_java_opts
Change #1248072 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper::server: if extra_java_opts are set, keep prometheus opts
Change #1248072 merged by Dzahn:
[operations/puppet@production] zookeeper::server: if extra_java_opts are set, keep prometheus opts
Change #1244944 merged by Dzahn:
[operations/puppet@production] zuul::main: add debugging extra_java_opts: "-Djavax.net.debug=ssl,handshake"
Change #1244927 merged by Dzahn:
[operations/puppet@production] zookeeper: support TLS by loading Netty jars into class path
Change #1244033 abandoned by Dzahn:
[operations/puppet@production] zuul::main: add extra Java opts to debug zookeeper TLS
Reason:
duplicate / replaced by other patches
Change #1248137 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: add zuul client cert to full chain of trust
Change #1248137 merged by Dzahn:
[operations/puppet@production] zuul::main: add zuul client cert to full chain of trust
Change #1248155 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: zuul scheduler needs to also have updated cert path
Change #1248155 merged by Dzahn:
[operations/puppet@production] zuul: zuul scheduler needs to also have updated cert path
Change #1260833 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use full chain as zookeeper TLS CA bundle
Change #1260833 merged by Dzahn:
[operations/puppet@production] zuul: use full chain as zookeeper TLS CA bundle
zuul-web now works (details in subtask!) :)
We have a new issue with zuul-scheduler (T421330).
It's similar to T405119#11752163
This appears to happen when multiple zuul services perform database migrations at the same time.
It seems we are supposed to:
Change #1261567 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: mariadb+pymysql instead of mysql+pymysql for DB connection
Change #1261567 merged by Dzahn:
[operations/puppet@production] zuul: mariadb+pymysql instead of mysql+pymysql for DB connection
Debugging the following issue we see on any zuul-related service that starts as the second service after the first one has connected to mariadb.
Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/local/lib/python3.11/dist-packages/alembic/script/base.py", line 415, in _upgr> Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: with self._catch_revision_errors( Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: self.gen.throw(typ, value, traceback) Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/local/lib/python3.11/dist-packages/alembic/script/base.py", line 253, in _catc> Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: raise util.CommandError(err.args[0]) from err Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: alembic.util.exc.CommandError: revision identifier b'6c1582c1d08c' is not a string; ensure > Mar 26 23:41:57 zuul1001 systemd[1]: zuul-web.service: Main process exited, code=exited, status=1/FAILURE
revision identifier b'6c1582c1d08c' is not a string
MariaDB [zuul]> describe alembic_version; +-------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+---------------+------+-----+---------+-------+ | version_num | varbinary(32) | NO | PRI | NULL | | +-------------+---------------+------+-----+---------+-------+ 1 row in set (0.003 sec)
So if i DROP that table and let zuul-web restart, it creates it again and again with varbinary type.
Then I tried adding ?charset=utf8mb4 to the dburi in the zuul config file.
Then tried altering the table to a varchar type. Without specifying a charset this was silently ignored.
But if you also set the charset you can ALTER it.
MariaDB [zuul]> ALTER TABLE alembic_version
-> MODIFY COLUMN version_num VARCHAR(32) NOT NULL;
Query OK, 0 rows affected (0.005 sec)
Records: 0 Duplicates: 0 Warnings: 0
MariaDB [zuul]> describe alembic_version;
+-------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| version_num | varbinary(32) | NO | PRI | NULL | |
+-------------+---------------+------+-----+---------+-------+
1 row in set (0.003 sec)
MariaDB [zuul]> ALTER TABLE alembic_version
-> MODIFY COLUMN version_num VARCHAR(32) CHARACTER SET utf8mb4 NOT NULL;
Query OK, 1 row affected (0.009 sec)
Records: 1 Duplicates: 0 Warnings: 0
MariaDB [zuul]> describe alembic_version;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| version_num | varchar(32) | NO | PRI | NULL | |
+-------------+-------------+------+-----+---------+-------+
1 row in set (0.003 sec)Now both zuul-scheduler and zuul-web could start.
Change #1261670 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: specify charset=utf8mb4 in database connection config
Change #1261670 merged by Dzahn:
[operations/puppet@production] zuul: specify charset=utf8mb4 in database connection config
Change #1261690 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: break out mTLS setup into separate class
regarding zuul-web: the service is running:
[zuul1001:~] $ systemctl status zuul-web
● zuul-web.service - zuul-web service
Loaded: loaded (/usr/lib/systemd/system/zuul-web.service; enabled; preset: enabled)
Active: active (running) since Sat 2026-03-28 14:36:54 UTC; 6 days ago
..
Apr 03 17:45:11 zuul1001 docker[2782311]: 2026-04-03 17:45:11,684 INFO zuul.ComponentRegistry: Noticed new scheduler component eca59e1df0bb0000000313
Apr 03 17:45:11 zuul1001 docker[2782311]: 2026-04-03 17:45:11,687 INFO zuul.ComponentRegistry: Component scheduler eca59e1df0bb0000000313 updated: {'hostname': 'eca59e1df0bb', 'kind': 'sche>I can connect to port 80 with curl to Apache httpd which is also running:
[zuul1001:~] $ curl zuul.discovery.wmnet <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> </body></html>
I can also connect directly to port 9000 where httpd proxies to:
telnet zuul.discovery.wmnet 9000 Trying 2620:0:861:103:10:64:32:104... Connected to zuul.discovery.wmnet. Escape character is '^]'. Connection closed by foreign host.
Still have to debug from there.
Change #1193142 abandoned by Dzahn:
[operations/puppet@production] zuul::main: add second zookeeper server to nodepool config (WIP)
Reason:
https://phabricator.wikimedia.org/T422207#11786389
Change #1260847 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::base: ensure /var/ssh/zuul exists
Change #1260847 merged by Dzahn:
[operations/puppet@production] zuul::base: ensure /var/ssh/zuul exists
Change #1269053 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::base: use wmflib::mkdir_p to ensure directories
Change #1269053 merged by Dzahn:
[operations/puppet@production] zuul::base: use wmflib::mkdir_p to ensure directories
Change #1269073 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: add TLS full chain needed for zookeeper config
Change #1269073 merged by Dzahn:
[operations/puppet@production] zuul::executor: add TLS full chain needed for zookeeper config
Change #1261690 abandoned by Dzahn:
[operations/puppet@production] zuul: break out mTLS setup into separate class
Reason:
solved in another way
Change #1269082 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: remove mounting of /etc/cfssl
Change #1269082 merged by Dzahn:
[operations/puppet@production] zuul::executor: remove mounting of /etc/cfssl
Change #1270103 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler
Change #1270103 merged by Dzahn:
[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler
Created a new ed25519 key pair for new zuul to connect to Gerrit (in the future).
[Ops] [puppet-private] (3334cf48f) (dzahn) add new ed25519 keypair for new zuul to connect to gerrit (T395938)
It lives under secrets/gerrit/zuul_gerrit_ed25519(.pub). Has NOT been added on the Gerrit side yet.
Change #1270577 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/private@master] add fake keys for new zuul to connect to gerrit
Change #1270577 merged by Dzahn:
[labs/private@master] add fake keys for new zuul to connect to gerrit
Change #1270580 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it
Change #1270580 merged by Dzahn:
[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it