The 6 VMs created for new zuul in T393873 need further setup / puppetization.
To avoid amending to the VM request ticket forever but have something to link changes to, continue on this new ticket.
This ticket covers further setup for all 3 types of new zuul VMs, main, executor and trusted job runner.
all changes in topic branch: https://gerrit.wikimedia.org/r/q/topic:%22zuul-new%22
Change #1244033 abandoned by Dzahn:
[operations/puppet@production] zuul::main: add extra Java opts to debug zookeeper TLS
Reason:
duplicate / replaced by other patches
Change #1248137 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::main: add zuul client cert to full chain of trust
Change #1248137 merged by Dzahn:
[operations/puppet@production] zuul::main: add zuul client cert to full chain of trust
Change #1248155 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: zuul scheduler needs to also have updated cert path
Change #1248155 merged by Dzahn:
[operations/puppet@production] zuul: zuul scheduler needs to also have updated cert path
Change #1260833 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use full chain as zookeeper TLS CA bundle
Change #1260833 merged by Dzahn:
[operations/puppet@production] zuul: use full chain as zookeeper TLS CA bundle
zuul-web now works (details in subtask!) :)
We have a new issue with zuul-scheduler (T421330).
It's similar to T405119#11752163
This appears to happen when multiple zuul services perform database migrations at the same time.
It seems we are supposed to:
Change #1261567 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: mariadb+pymysql instead of mysql+pymysql for DB connection
Change #1261567 merged by Dzahn:
[operations/puppet@production] zuul: mariadb+pymysql instead of mysql+pymysql for DB connection
Debugging the following issue we see on any zuul-related service that starts as the second service after the first one has connected to mariadb.
Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/local/lib/python3.11/dist-packages/alembic/script/base.py", line 415, in _upgr> Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: with self._catch_revision_errors( Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: self.gen.throw(typ, value, traceback) Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: File "/usr/local/lib/python3.11/dist-packages/alembic/script/base.py", line 253, in _catc> Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: raise util.CommandError(err.args[0]) from err Mar 26 23:41:56 zuul1001 docker[2350294]: 2026-03-26 23:41:56,910 ERROR zuul.WebServer: alembic.util.exc.CommandError: revision identifier b'6c1582c1d08c' is not a string; ensure > Mar 26 23:41:57 zuul1001 systemd[1]: zuul-web.service: Main process exited, code=exited, status=1/FAILURE
revision identifier b'6c1582c1d08c' is not a string
MariaDB [zuul]> describe alembic_version; +-------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+---------------+------+-----+---------+-------+ | version_num | varbinary(32) | NO | PRI | NULL | | +-------------+---------------+------+-----+---------+-------+ 1 row in set (0.003 sec)
So if i DROP that table and let zuul-web restart, it creates it again and again with varbinary type.
Then I tried adding ?charset=utf8mb4 to the dburi in the zuul config file.
Then tried altering the table to a varchar type. Without specifying a charset this was silently ignored.
But if you also set the charset you can ALTER it.
MariaDB [zuul]> ALTER TABLE alembic_version
-> MODIFY COLUMN version_num VARCHAR(32) NOT NULL;
Query OK, 0 rows affected (0.005 sec)
Records: 0 Duplicates: 0 Warnings: 0
MariaDB [zuul]> describe alembic_version;
+-------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| version_num | varbinary(32) | NO | PRI | NULL | |
+-------------+---------------+------+-----+---------+-------+
1 row in set (0.003 sec)
MariaDB [zuul]> ALTER TABLE alembic_version
-> MODIFY COLUMN version_num VARCHAR(32) CHARACTER SET utf8mb4 NOT NULL;
Query OK, 1 row affected (0.009 sec)
Records: 1 Duplicates: 0 Warnings: 0
MariaDB [zuul]> describe alembic_version;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| version_num | varchar(32) | NO | PRI | NULL | |
+-------------+-------------+------+-----+---------+-------+
1 row in set (0.003 sec)Now both zuul-scheduler and zuul-web could start.
Change #1261670 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: specify charset=utf8mb4 in database connection config
Change #1261670 merged by Dzahn:
[operations/puppet@production] zuul: specify charset=utf8mb4 in database connection config
Change #1261690 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: break out mTLS setup into separate class
regarding zuul-web: the service is running:
[zuul1001:~] $ systemctl status zuul-web
● zuul-web.service - zuul-web service
Loaded: loaded (/usr/lib/systemd/system/zuul-web.service; enabled; preset: enabled)
Active: active (running) since Sat 2026-03-28 14:36:54 UTC; 6 days ago
..
Apr 03 17:45:11 zuul1001 docker[2782311]: 2026-04-03 17:45:11,684 INFO zuul.ComponentRegistry: Noticed new scheduler component eca59e1df0bb0000000313
Apr 03 17:45:11 zuul1001 docker[2782311]: 2026-04-03 17:45:11,687 INFO zuul.ComponentRegistry: Component scheduler eca59e1df0bb0000000313 updated: {'hostname': 'eca59e1df0bb', 'kind': 'sche>I can connect to port 80 with curl to Apache httpd which is also running:
[zuul1001:~] $ curl zuul.discovery.wmnet <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> </body></html>
I can also connect directly to port 9000 where httpd proxies to:
telnet zuul.discovery.wmnet 9000 Trying 2620:0:861:103:10:64:32:104... Connected to zuul.discovery.wmnet. Escape character is '^]'. Connection closed by foreign host.
Still have to debug from there.
Change #1193142 abandoned by Dzahn:
[operations/puppet@production] zuul::main: add second zookeeper server to nodepool config (WIP)
Reason:
https://phabricator.wikimedia.org/T422207#11786389
Change #1260847 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::base: ensure /var/ssh/zuul exists
Change #1260847 merged by Dzahn:
[operations/puppet@production] zuul::base: ensure /var/ssh/zuul exists
Change #1269053 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::base: use wmflib::mkdir_p to ensure directories
Change #1269053 merged by Dzahn:
[operations/puppet@production] zuul::base: use wmflib::mkdir_p to ensure directories
Change #1269073 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: add TLS full chain needed for zookeeper config
Change #1269073 merged by Dzahn:
[operations/puppet@production] zuul::executor: add TLS full chain needed for zookeeper config
Change #1261690 abandoned by Dzahn:
[operations/puppet@production] zuul: break out mTLS setup into separate class
Reason:
solved in another way
Change #1269082 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: remove mounting of /etc/cfssl
Change #1269082 merged by Dzahn:
[operations/puppet@production] zuul::executor: remove mounting of /etc/cfssl
Change #1270103 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler
Change #1270103 merged by Dzahn:
[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler
Created a new ed25519 key pair for new zuul to connect to Gerrit (in the future).
[Ops] [puppet-private] (3334cf48f) (dzahn) add new ed25519 keypair for new zuul to connect to gerrit (T395938)
It lives under secrets/gerrit/zuul_gerrit_ed25519(.pub). Has NOT been added on the Gerrit side yet.
Change #1270577 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/private@master] add fake keys for new zuul to connect to gerrit
Change #1270577 merged by Dzahn:
[labs/private@master] add fake keys for new zuul to connect to gerrit
Change #1270580 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it
Change #1270580 merged by Dzahn:
[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it
Change #1275964 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add new public key for zuul <-> gerrit 2026
Change #1275964 merged by Dzahn:
[operations/puppet@production] zuul: add new public key for zuul <-> gerrit 2026
Since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275972 added a firewall rule the executors can now talk to zookeeper on the main machines.
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1178084 has been merged which created
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277195 parameterized web root and host
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198 switched to using the master branch
after purging the URL from CDN caches we can now see at least:
view-source:https://zuul.wikimedia.org/
(view the source)
Change #1277771 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] cache: add normal caching setting for zuul.wikimedia.org
Change #1277771 merged by Dzahn:
[operations/puppet@production] cache: add normal caching setting for zuul.wikimedia.org
@dduvall after merging the additional change above to set "caching: normal" I now see content:
Change #1271042 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: allow zuul machines to port 22 ssh (WIP)
Change #1278534 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: switch to new discovery2026 intermediate CA
Change #1278534 merged by Dzahn:
[operations/puppet@production] zuul: switch to new discovery2026 intermediate CA
Change #1279470 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: create profile for new zuul-builder replacing nodepool
Change #1279461 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: remove zuul-nodepool config, user, stop service
Change #1279461 merged by Dzahn:
[operations/puppet@production] zuul: remove zuul-nodepool config, user, stop service
zuul-nodepool has been removed from zuul::main machines. (config, systemd unit, docker container not running, etc)
Change #1280729 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/private@master] zuul: rename nodepool::user_token to launcher::user_token
Change #1280729 merged by Dzahn:
[labs/private@master] zuul: rename nodepool::user_token to launcher::user_token
Change #1279470 merged by Dzahn:
[operations/puppet@production] zuul: create profile for new zuul-launcher replacing nodepool
Change #1280815 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: remove nodepool profile from zuul::main role
Change #1280815 merged by Dzahn:
[operations/puppet@production] zuul: remove nodepool profile from zuul::main role
Change #1280820 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add placeholder template for launcher config
Change #1280820 merged by Dzahn:
[operations/puppet@production] zuul: add placeholder template for launcher config
Change #1280829 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it
Change #1280829 merged by Dzahn:
[operations/puppet@production] zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it
Change #1280832 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: remove nodepool-related code
Change #1280832 merged by Dzahn:
[operations/puppet@production] zuul: remove nodepool-related code
Change #1280840 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: update zuul-launcher version to 14.2.0-1
Change #1280840 merged by Dzahn:
[operations/puppet@production] zuul: update zuul-launcher version to 14.2.0-1
Change #1281595 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add launcher_connection Hiera key to executor role
Change #1281595 merged by Dzahn:
[operations/puppet@production] zuul: add launcher_connection Hiera key to executor role
Change #1271042 abandoned by Dzahn:
[operations/puppet@production] gerrit: allow zuul machines to port 22 ssh
Change #1287035 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: make all service_ensures dependent on a single active server
Change #1287483 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: disable all services in codfw, keep enabled in eqiad
Change #1287035 abandoned by Dzahn:
[operations/puppet@production] zuul: make all service_ensures dependent on a single active server
Reason:
in favor of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1287483 might restore later
Change #1287483 merged by Dzahn:
[operations/puppet@production] zuul: disable all services in codfw, keep enabled in eqiad
Change #1287933 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: let the launcher use the zuul user, not a separate one
Change #1287933 merged by Dzahn:
[operations/puppet@production] zuul: let the launcher use the zuul user, not a separate one
Change #1286999 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: replace user/group setup with systemd::sysuser
Change #1286999 merged by Dzahn:
[operations/puppet@production] zuul: replace user/group setup with systemd::sysuser
@dduvall Currently I don't have any open patches here anymore. Got any opinions on what is missing on the puppet side now?
Since we celebrated the first succesful build on new zuul - I am optimistically calling this resolved too since the underlying infra has been setup.