zuul-web serves the web dashboard to end users Communicates with Database, Zookeeper.
This will be a systemd unit using the zuul-web image: docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-web:wmf-12.0.0-5
Per @Corvus in T395938#10929023:
Bind mounts for:
And:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T395938 puppetize setup of new zuul VMs | |||
| Resolved | Dzahn | T405119 Set up zuul web on zuul1001/zuul2001 | |||
| Resolved | Dzahn | T406938 SystemdUnitFailed - zuul-web.service on zuul1001:9100 | |||
| Resolved | Dzahn | T407303 SystemdUnitFailed - zuul-web.service on zuul1001:9100 |
Change #1194306 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: create class and systemd unit for new zuul-web service
Change #1194306 merged by Dzahn:
[operations/puppet@production] zuul: create class and systemd unit for new zuul-web service
[zuul1001:~] $ sudo systemctl status zuul-web
● zuul-web.service - zuul-web service
Loaded: loaded (/lib/systemd/system/zuul-web.service; enabled; preset: enabled)
Active: active (running) since Wed 2025-10-08 17:48:50 UTC; 5min ago
Main PID: 3571531 (docker)
Tasks: 9 (limit: 9468)
Memory: 44.6M
CPU: 50ms
CGroup: /system.slice/zuul-web.service
└─3571531 /usr/bin/docker run --mount type=bind,src=/etc/zuul,dst=/etc/zuul --mount type=bind,src=/etc/cfssl,dst=/etc/cfssl docker-registry.wikimedia.org/repos/releng/zuul/zuul>
Oct 08 17:48:51 zuul1001 docker[3571531]: 7ae311faa028: Already existsChange #1194710 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: run zuul-web services as zuul user
Change #1194710 merged by Dzahn:
[operations/puppet@production] zuul: run zuul-web services as zuul user
Change #1194719 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add port mapping for port 9000 for zuul-web service
Change #1194719 merged by Dzahn:
[operations/puppet@production] zuul: add port mapping for port 9000 for zuul-web service
The zuul-web service and container is running. Port 9000 should be mapped to the host.
I just don't see the container having a process that listens on port 9000 (by running netstat inside it).
Change #1195053 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add host network to docker command for new zuul-web component
Change #1195053 merged by Dzahn:
[operations/puppet@production] zuul: add host network to docker command for new zuul-web component
Change #1195062 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add missing $host_ip variable to zuul-web class
Change #1195062 merged by Dzahn:
[operations/puppet@production] zuul: add missing $host_ip variable to zuul-web class
zuul1001:/etc/apache2/sites-enabled] $ telnet localhost 9000 Trying ::1... Connected to localhost. Escape character is '^]'.
mostly works..just:
curl --resolve 'zuul.wikimedia.org:80:10.64.32.104' http://zuul.wikimedia.org/zuul/status <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> </body></html>
Change #1195067 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] httpbb: add minimal tests for new zuul-web
Change #1195067 merged by Dzahn:
[operations/puppet@production] httpbb: add minimal tests for new zuul-web
Change #1195073 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] httpbb: add missing directory for new zuul tests
Change #1195073 merged by Dzahn:
[operations/puppet@production] httpbb: add missing directory for new zuul tests
Change #1195347 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: add firewall rule to allow zuul-web to httpd
Change #1195347 merged by Dzahn:
[operations/puppet@production] zuul: add firewall rule to allow zuul-web to httpd
more debugging and the remaining issues we are experiencing probably all come down to the zookeeper connection. or rather:
2025-10-15 00:08:28,451 - WARN [NIOWorkerThread-4:NIOServerCnxn@383] - Close of session 0x0
java.io.IOException: Len error. A message from /172.17.0.4:49048 with advertised length of 369295618 is either a malformed message or too large to process (length is greater than jute.maxbuffer=1048575)
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:552)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:342)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)@Dzahn Looking at the zuul-web container logs, it seems like the zookeeper connection is failing outright.
dduvall@zuul1001:~$ sudo docker logs brave_chatelet | tail -n 3 2025-10-27 20:06:44,223 WARNING zuul.zk.base.ZooKeeperClient: Retrying zookeeper connection 2025-10-27 20:06:54,653 WARNING zuul.zk.base.ZooKeeperClient: Retrying zookeeper connection 2025-10-27 20:07:05,075 WARNING zuul.zk.base.ZooKeeperClient: Retrying zookeeper connection
After diving into the zuul.zk client code a bit, I was able to create a minimal reproduction of the problem in python.
dduvall@zuul1001:~$ sudo docker exec -it brave_chatelet /usr/bin/env -- python3
Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kazoo.client import KazooClient
>>>
>>>
>>> client = KazooClient(
... hosts=["10.64.32.104:2181"],
... use_ssl=True,
... keyfile="/etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet-key.pem",
... certfile="/etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet.pem",
... ca="/etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet.chain.pem",
... )
>>> client.start()
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
^CTraceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.11/dist-packages/kazoo/client.py", line 630, in start
event.wait(timeout=timeout)
File "/usr/lib/python3.11/threading.py", line 622, in wait
signaled = self._cond.wait(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/threading.py", line 324, in wait
gotit = waiter.acquire(True, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
>>> Connection dropped: socket connection error: TLS/SSL connection has been closed (EOF) (_ssl.c:992)Going even more minimal using just openssl s_client and the host/key/cert/ca I get the same EOF related error.
dduvall@zuul1001:~$ sudo openssl s_client -connect 10.64.32.104:2181 -cert /etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet.pem -key /etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet-key.pem -CAfile /etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zuul__zuul1001_eqiad_wmnet.chain.pem Connecting to 10.64.32.104 CONNECTED(00000003) 4027E76A707F0000:error:0A000126:SSL routines::unexpected eof while reading:../ssl/record/rec_layer_s3.c:696: --- no peer certificate available --- No client certificate CA names sent Negotiated TLS1.3 group: <NULL> --- SSL handshake has read 0 bytes and written 1540 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Protocol: TLSv1.3 This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
Perhaps you can use the above openssl s_client command and see whether there are corresponding logs on the server end regarding some TLS failure.
Disabling TLS appears to work, so it seems the server does not actually have TLS enabled. :)
dduvall@zuul1001:~$ sudo docker exec -it brave_chatelet /usr/bin/env -- python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from kazoo.client import KazooClient >>> >>> >>> client = KazooClient( ... hosts=["10.64.32.104:2181"], ... use_ssl=False, ... ) >>> client.start() >>> client.command(b"ruok") 'imok' >>> quit()
@dduvall Thanks for looking! Sorry, I should have made this more obvious.. but: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197339
The change above is to add TLS support to zookeeper.. which I created once I got to the same place during debugging.
The other alternative is .. to not try to use TLS. I mean.. none of the other zookeeper servers in WMF prod do it.. as evidenced by the need to add support for it.
And we are not leaving our VM with this traffic.
The Zuul/ZooKeeper docs suggest that TLS connections are required and indeed it seems to be enforced in code.
Looks like this one merged, and I see there's now a keystore zookeeper_zuul.keystore.p12. Zookeeper restarted about a week ago and I can see it's using an updated config. But zuul web is still having trouble connecting. Using openssl s_client -connect I'm seeing the same unexpected eof that @dduvall got back in Oct. Unsure I can investigate much further without knowing the keystore password. Any ideas @Dzahn ?
Come to think of it...how is zookeeper able to use the keystore without the password? 😅 I didn't see reference to the password in the zookeeper config or the java runtime options—is that the problem? Looks like zookeeper supports an ssl.keyStore.passwordPath.
Change #1224908 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: add ssl.keyStore.passwordPath is TLS is enabled (WIP)
You have a good point there, Tyler. Thank you! I have started looking and made a reminder WIP patch to add the passwordPath. Will get to it after the weekend.
Change #1227735 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: write TLS passphrase to a file for zookeeper
Change #1227735 merged by Dzahn:
[operations/puppet@production] zuul: write TLS passphrase to a file for zookeeper
Change #1233697 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: add parameter and path to tls cert passphrase
Change #1233697 abandoned by Dzahn:
[operations/puppet@production] zookeeper: add parameter and path to tls cert passphrase
Reason:
merge into https://gerrit.wikimedia.org/r/c/operations/puppet/+/1224908
Change #1224908 merged by Dzahn:
[operations/puppet@production] zookeeper: add ssl.keyStore.passwordPath
@thcipriani Hi from the SRE summit. With the merges above there is now a file with the keystore passphrase and the zookeeper config has ssl.keyStore.passwordPath=/etc/zookeeper/conf/zuul_tls and restarted zookeeper and zuul-web.
Did not get to debug it more yet, but seems like there is still an issue.
@Dzahn After debugging for a bit with @thcipriani I finally just looked at the strace of zookeeper and it appears that zookeeper doesn't have permission to read files beneath /etc/cfssl:
[pid 2456391] openat(AT_FDCWD, "/etc/cfssl/ssl/zuul__zuul1001_eqiad_wmnet/zookeeper_zuul.keystore.p12", O_RDONLY) = -1 EACCES (Permission denied)
dduvall@zuul1001:~$ ls -ld /etc/cfssl/ dr-xr-x--- 6 root root 4096 Oct 20 18:29 /etc/cfssl/
In addition to fixing the permissions issue, it would be great to get zookeeper logging working. It doesn't seem to be logging anything to /var/log/zookeeper and there's isn't much from journalctl -fu zookeeper because the service definition is wrapping an old sysvinit script (/etc/init.d/zookeeper), so we just see the started/stopped messages and that's it.
Thank you! Ack, confirmed. Oof, the permission problem is not as trivial as it may sound since we don't want to change the behaviour of the cfssl module that is used in many other places and the "add existing user to existing group" problem in puppet. Thinking about it.
Oooh, yay:) modules/cfsssl actually already DOES have parameters for the owner:
# @param owner the user to use as the owner of files # @param group the user to use as the owner of files
BUT .. have to see how to pass it through from profile::pki::get_cert('zuul') to cfssl.
edit: We also already are doing that when we convert the cert to the keystore format.
sslcert::x509_to_pkcs12 { 'zookeeper_zuul_keystore' :
owner => 'zookeeper',
group => 'zookeeper',4.0K -r--r----- 1 zookeeper zookeeper 2.4K Jan 30 16:14 zookeeper_zuul.keystore.p12 4.0K -rw-r--r-- 1 root root 2.4K Jan 30 15:44 zuul__zuul1001_eqiad_wmnet.chained.pem
so the issue is the parent directory.
Change #1236386 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: set owner and notify zookeeper service with pki::get_cert
Change #1236390 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: move cert paths to role level, drop host-name based config
Change #1236386 merged by Dzahn:
[operations/puppet@production] zuul: set owner and notify zookeeper service with pki::get_cert
Change #1236390 merged by Dzahn:
[operations/puppet@production] zuul: move cert paths to role level, drop host-name based config
Change #1236730 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper/zuul: use standard port 2281 for TLS secureClientPort
Change #1236735 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zookeeper: set keystore format to PKCS12 when enabling TLS (for zuul)
Change #1236730 merged by Dzahn:
[operations/puppet@production] zookeeper/zuul: use standard port 2281 for TLS secureClientPort
Change #1236735 merged by Dzahn:
[operations/puppet@production] zookeeper: set keystore format to PKCS12 when enabling TLS (for zuul)
Change #1236809 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: move .p12 keystore file under the zookeeper config path
Change #1236809 merged by Dzahn:
[operations/puppet@production] zuul: move .p12 keystore file under the zookeeper config path
Change #1236815 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use chained certificate incl CA for zookeeper
Change #1236815 merged by Dzahn:
[operations/puppet@production] zuul: use chained certificate incl CA for zookeeper
@Dzahn circling back to this. Thanks for pushing forward with it!
Still seeing this currently:
dduvall@zuul1001:~$ sudo journalctl -u zuul-web -n 1 Mar 24 19:04:39 zuul1001 docker[735]: 2026-03-24 19:04:39,804 WARNING zuul.zk.base.ZooKeeperClient: Retrying zookeeper connection
And this when attempting to connect via kazoo.KazooClient:
Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from kazoo.client import KazooClient >>> client = KazooClient(hosts=["10.64.32.104:2281"], keyfile="/etc/zookeeper/zuul-tls/zuul__zuul-key.pem", certfile="/etc/zookeeper/zuul-tls/zuul__zuul.pem", ca="/etc/zookeeper/zuul-tls/zuul__zuul.chain.pem", use_ssl=True) >>> client.start() Connection dropped: socket connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:992)
However, if I pass ca="/etc/zookeeper/zuul-tls/zuul_full_chain.pem", it works!
>>> client = KazooClient(hosts=["10.64.32.104:2281"], keyfile="/etc/zookeeper/zuul-tls/zuul__zuul-key.pem", certfile="/etc/zookeeper/zuul-tls/zuul__zuul.pe m", ca="/etc/zookeeper/zuul-tls/zuul_full_chain.pem", use_ssl=True) >>> client.start() >>> client.command(b"ruok") 'imok'
I'll submit a change to operations/puppet to have it use zuul_full_chain.pem for the CA cert.
Thanks, but it already works. The cert issues were solved. But what we need is a wipe of the database.
Multiple issues before were related to certs and chains and multiple fixes were deployed including building a full chain that includes cert, intermediate zuul CA and WMF puppet CA. Notably also https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236815/1/modules/profile/manifests/zuul/main.pp
Then I finally had it working. And ran into an unrelated issue with contents in the database. Which can only happen once past the cert issue.
Yeah, I was looking at those patches. Why do I still see connection failures in the zuul-web logs?
Change #1260833 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use full chain as zookeeper TLS CA bundle
Thanks, I am very happy it finally works.
However, if I pass ca="/etc/zookeeper/zuul-tls/zuul_full_chain.pem", it works!
Yes! I can confirm that. Getting to the point where we have that "full_chain" file was the journey here.
We do not get that file by default from the pki::get_cert class and most things don't require it.
But Java (Zookeeper) here is more picky.
I added code around here, with these concat::fragments to build the full chain:
The full chain is also already configured in Hiera.
What was missing was just that last step to change the config in the erb template.
So I had this working and then puppet was re-enabled again and reverted that.
This is why I was like "it already works" but then confused at first.
I'll submit a change to operations/puppet to have it use zuul_full_chain.pem for the CA cert.
Thanks! As per our IRC chat just now I made
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260833
Per our IRC discussion, it may be a good idea long-term to refactor profile::pki::get_cert and the underlying cfssl::cert resource to optionally include the entire CA bundle including the root CA.
Mentioned in SAL (#wikimedia-operations) [2026-03-25T23:29:11Z] <mutante> zuul1001 - installed mariadb-client - connected once to zuul db on m1-master; mysql> truncate "alembic_version"; - systemctl restart zuul-web - This fixed the zuul-web service. finally no error in systemctl status. (T405119)
After the cert issues were fixed in puppet we got to that database error I mentioned before, that I had seen last time I had it working via manual config change:
Mar 25 23:09:56 zuul1001 docker[2311608]: 2026-03-25 23:09:56,733 ERROR zuul.WebServer: alembic.util.exc.CommandError: revision identifier b'6c1582c1d08c' is not a string; ensure database ...
So I installed mariadb client; connected to the zuul database and did a mysql> truncate alembic_version;.
Then after a final systemctl restart zuul-web it now finally works! :)
root@zuul1001:/home/dzahn# systemctl status zuul-web
● zuul-web.service - zuul-web service
Loaded: loaded (/usr/lib/systemd/system/zuul-web.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-03-25 23:26:06 UTC; 4min 10s ago
Invocation: 719ef19fc9ec41d58e9ecca413e79738
Main PID: 2318116 (docker)
Tasks: 9 (limit: 9469)
Memory: 12.4M (peak: 12.8M)
CPU: 102ms
...
Mar 25 23:27:09 zuul1001 docker[2318116]: 2026-03-25 23:27:09,659 INFO zuul.GerritConnection: Creating Zookeeper event queue
Mar 25 23:27:09 zuul1001 docker[2318116]: 2026-03-25 23:27:09,666 INFO zuul.WebServer: Zuul Web Server starting
Mar 25 23:27:09 zuul1001 docker[2318116]: 2026-03-25 23:27:09,666 INFO zuul.web: ZuulWeb starting
Mar 25 23:27:09 zuul1001 docker[2318116]: 2026-03-25 23:27:09,680 INFO zuul.web: Starting command processor
Mar 25 23:27:09 zuul1001 docker[2318116]: 2026-03-25 23:27:09,683 INFO zuul.web: Waiting for system config from schedulerYes, there are ERROR zuul.GerritConnection lines but at this point that is intentional. We don't want it to talk to Gerrit right now.