Summary of the above:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T395938 puppetize setup of new zuul VMs | |||
| Resolved | Dzahn | T403847 Deploy zuul executor on executor VM | |||
| Resolved | Dzahn | T404478 SystemdUnitFailed - zuul-executor | |||
| Open | dduvall | T424061 Ensure zuul-executor can prepare workspaces and dispatch builds | |||
| Resolved | Dzahn | T421232 zuul1002 fails to compile in PCC | |||
| Resolved | Dzahn | T406366 SystemdUnitFailed - zuul-executor |
[zuul1002:/] $ sudo file /etc/zuul/ssh/id_rsa /etc/zuul/ssh/id_rsa: OpenSSH private key
Change #1185174 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: define main and executor host names in common hiera data
Change #1185174 merged by Dzahn:
[operations/puppet@production] zuul: define main and executor host names in common hiera data
Change #1185180 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: firewall rule to allow main nodes to zuul-web port
Change #1185180 merged by Dzahn:
[operations/puppet@production] zuul::executor: firewall rule to allow main nodes to zuul-web port
Needs to allow connections from the zuul-web/fingergateway node on port 7900
zuul1002:~] $ sudo iptables -L | grep 7900 ACCEPT tcp -- zuul2001.codfw.wmnet anywhere tcp dpt:7900 ACCEPT tcp -- zuul1001.eqiad.wmnet anywhere tcp dpt:7900
Change #1185184 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: use profile::docker::engine to install docker
Change #1185184 merged by Dzahn:
[operations/puppet@production] zuul::executor: use profile::docker::engine to install docker
Change #1185196 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: ensure /var/lib/zuul dir exists
Change #1185196 merged by Dzahn:
[operations/puppet@production] zuul::executor: ensure /var/lib/zuul dir exists
We need an /etc/zuul/zuul.conf specific to executors.
--> https://zuul-ci.org/docs/zuul/latest/configuration.html#id5
executor.manage_ansible
It is recommended to set this to False and manually install Ansible after the Zuul installation by running zuul-manage-ansible.
We can't really do either of these things in production.
The Zuul executor will run from a Docker image which already has Ansible installed:
$ docker run --rm -it --entrypoint=zuul-manage-ansible docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-executor:wmf-12.0.0-5 -l 8 9 (default)
In the images, they are under /usr/local/lib/zuul/ansible.
I see this config file in the zuul::main profile you set up. It looks to have the executor section which looks right. Need to ensure that (or part of that?) config file is on executors and that the path to the ssh key is correct for the bind mounts for the executor image.
Yes, let me try the simplest option first and ensure the same config file is deployed on both machines.
Change #1186045 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: add zuul config file
Change #1186045 merged by Dzahn:
[operations/puppet@production] zuul::executor: add zuul config file
Next step to debug is:
2025-09-08 20:01:28,869 ERROR zuul.BubblewrapDriver: Non zero return code
..
File "/usr/local/lib/python3.11/dist-packages/zuul/driver/bubblewrap/__init__.py", line 224, in __init__
raise Exception('bwrap execution validation failed. You can '
Exception: bwrap execution validation failed. You can use `zuul-bwrap /tmp id` to investigate manually.where are you seeing that error? At a guess this might have to do with needing privileged mode, without that it's probably not able to effectively "wrap" things.
[zuul1002:~] $ sudo docker tag 4ba8235d2c0b executor .. [zuul1002:~] $ sudo docker run --privileged --mount type=bind,src=/etc/zuul,dst=/etc/zuul executor
It did not change between adding --privileged or not.
I am going to try to reduce the config file to just the executor part.
Change #1186567 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul(new): update path to ssh private key for zuul executor
Change #1186567 merged by Dzahn:
[operations/puppet@production] zuul(new): update path to ssh private key for zuul executor
running bubblewrap directly (zuul-wrap /tmp id mentioned in error message to debug)
[zuul1002:~] $ sudo docker run --privileged --mount type=bind,src=/etc/zuul,dst=/etc/zuul executor zuul-bwrap /tmp id
DEBUG:zuul.BubblewrapDriver:Initializing bubblewrap with oom_score_adj starting: 0, final: 200
DEBUG:zuul.BubblewrapExecutionContext:Bubblewrap command: setpriv --ambient-caps -all choom -n 200 -- bwrap --dir /tmp --tmpfs /tmp --dir /var --dir /var/tmp --dir /run/user/0 --ro-bind /usr /usr --ro-bind /lib /lib --ro-bind /bin /bin --ro-bind /sbin /sbin --ro-bind /etc/ld.so.cache /etc/ld.so.cache --ro-bind /etc/resolv.conf /etc/resolv.conf --ro-bind /etc/hosts /etc/hosts --ro-bind /etc/localtime /etc/localtime --ro-bind /dev/null /dev/null --bind /tmp /tmp --tmpfs /tmp/tmp --proc /proc --dev /dev --chdir /tmp --unshare-all --share-net --die-with-parent --uid 0 --gid 0 --file 3 /etc/passwd --file 4 /etc/group --unshare-user --disable-userns --ro-bind /lib64 /lib64 --ro-bind /etc/nsswitch.conf /etc/nsswitch.conf --ro-bind /etc/alternatives /etc/alternatives --ro-bind /etc/ssl/certs /etc/ssl/certs --ro-bind /etc/subuid /etc/subuid --ro-bind /etc/containers /etc/containers
ERROR:zuul.BubblewrapDriver:Non zero return code executing: setpriv --ambient-caps -all choom -n 200 -- bwrap --dir /tmp --tmpfs /tmp --dir /var --dir /var/tmp --dir /run/user/0 --ro-bind /usr /usr --ro-bind /lib /lib --ro-bind /bin /bin --ro-bind /sbin /sbin --ro-bind /etc/ld.so.cache /etc/ld.so.cache --ro-bind /etc/resolv.conf /etc/resolv.conf --ro-bind /etc/hosts /etc/hosts --ro-bind /etc/localtime /etc/localtime --ro-bind /dev/null /dev/null --bind /tmp /tmp --tmpfs /tmp/tmp --proc /proc --dev /dev --chdir /tmp --unshare-all --share-net --die-with-parent --uid 0 --gid 0 --file 3 /etc/passwd --file 4 /etc/group --unshare-user --disable-userns --ro-bind /lib64 /lib64 --ro-bind /etc/nsswitch.conf /etc/nsswitch.conf --ro-bind /etc/alternatives /etc/alternatives --ro-bind /etc/ssl/certs /etc/ssl/certs --ro-bind /etc/subuid /etc/subuid --ro-bind /etc/containers /etc/containers id
Traceback (most recent call last):
File "/usr/local/bin/zuul-bwrap", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/dist-packages/zuul/driver/bubblewrap/__init__.py", line 353, in main
driver = BubblewrapDriver(check_bwrap=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/zuul/driver/bubblewrap/__init__.py", line 224, in __init__
raise Exception('bwrap execution validation failed. You can '
Exception: bwrap execution validation failed. You can use `zuul-bwrap /tmp id` to investigate manually."On systems with AppArmor (common on Ubuntu), it may be restricting bwrap's ability to create namespaces, even with the --privileged flag. You can try disabling AppArmor restrictions for unprivileged user namespaces."
Yea, we have apparmor installed. Testing that.
"Option 2: Add --disable-userns to the bwrap command"
And this.
I tried:
There is also a difference between bwrap and zuul-bwrap.
bwrap is version 0.8.0. ("Zuul versions 0.8.0 and newer are designed to detect if the host is using user namespaces and automatically add the --disable-userns flag to the bwrap command")
zuul-bwrap always fails with the Exception: bwrap execution validation failed. even with just --version.
bwrap can be executed.
Checked if this also happens on trixie with newer kernel version and on cloud VPS where production defaults for docker settings and kernel parameters don't apply.
Same issue.
Played around a bit with this. It looks like zuul-bwrap is trying to use bwrap.
I found a minimal repro case: bwrap --unshare-user --disable-userns --dev-bind / / /usr/bin/id
Docs say that creates a new user namespace then prevents sub user namespaces.
I straced it:
root@cb13e865c29b:/# strace -f -e trace=unshare,clone,setns,prctl bwrap --unshare-user --disable-userns --dev-bind / / /usr/bin/id 2>&1 ...blah blah blah... clone(child_stack=NULL, flags=CLONE_NEWNS|CLONE_NEWUSER|SIGCHLDstrace: Process 1240 attached ) = 1240 [pid 1239] prctl(PR_SET_DUMPABLE, SUID_DUMP_USER) = 0 [pid 1240] prctl(PR_CAPBSET_DROP, CAP_PERFMON) = 0 [pid 1240] prctl(PR_CAPBSET_DROP, CAP_BPF) = 0 [pid 1240] prctl(PR_CAPBSET_DROP, CAP_CHECKPOINT_RESTORE) = 0 [pid 1240] unshare(CLONE_NEWUSER) = -1 EPERM (Operation not permitted) bwrap: unshare user ns: Operation not permitted [pid 1240] +++ exited with 1 +++ +++ exited with 1 +++
So it creates a new usernamespace using clone. Then when it tries to unshare in that namespace, it fails.
I kinda wonder if it's:
thcipriani@zuul1002:~$ cat /proc/sys/kernel/unprivileged_userns_clone 1
which is different on my localhost vs. in this vm. I don't know, @dduvall do you remember anything like this when you were setting up a zuul executor in WMCS?
Found what makes a difference!
sudo sysctl -w kernel.unprivileged_userns_clone=1
makes the bwrap execution error go away!
Of the other options I tried above only --privileged is still needed.
Change #1186630 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: include profile::pki::client
Change #1186630 merged by Dzahn:
[operations/puppet@production] zuul::executor: add TLS certs for zookeeper config
sudo docker run --privileged --mount type=bind,src=/etc/zuul,dst=/etc/zuul --mount type=bind,src=/var/lib/zuul,dst=/var/lib/zuul --mount type=bind,src=/etc/cfssl,dst=/etc/cfssl executor ... 2025-09-09 23:12:40,676 INFO zuul.Executor: Starting log streamer
:)
Just have to puppetize the correct path to the zookeeper TLS certs.. since it includes the host name and therefore changes on each machine.
But after I manually changed the path to the certs and added another bind mount for /etc/cfssl ... executor starts !
Change #1187055 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: systctl setting unprivileged_userns_clone needed
Change #1187076 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: use variables to set path to zookeeper TLS certs in config
Change #1187076 merged by Dzahn:
[operations/puppet@production] zuul: use variables to set path to zookeeper TLS certs in config
Change #1187055 merged by Dzahn:
[operations/puppet@production] zuul::executor: systctl setting unprivileged_userns_clone needed
Change #1187495 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: use sysctl setting from base module, remove local code
Change #1187495 merged by Dzahn:
[operations/puppet@production] zuul::executor: use sysctl setting from base module, remove local code
Change #1187531 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: create systemd service
Change #1187531 merged by Dzahn:
[operations/puppet@production] zuul::executor: create systemd service
Change #1187899 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: fix name of systemd service template
Change #1187899 merged by Dzahn:
[operations/puppet@production] zuul::executor: fix name of systemd service template
Change #1187901 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: fix service description and path in systemd unit
Change #1187901 merged by Dzahn:
[operations/puppet@production] zuul::executor: fix service description and path in systemd unit
Change #1188844 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: hiera'ize docker image version and use in template
Change #1188844 merged by Dzahn:
[operations/puppet@production] zuul::executor: hiera'ize docker image version and use in template
Change #1188907 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: drop bootstrap file, adjust systemd service command line
Change #1188907 merged by Dzahn:
[operations/puppet@production] zuul::executor: drop bootstrap file, adjust systemd service command line
Change #1188911 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: configure exit code 143 as success in systemd unit
Change #1188911 merged by Dzahn:
[operations/puppet@production] zuul::executor: configure exit code 143 as success in systemd unit
Change #1189604 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul::executor: let executor connect to zookeeper on the host machine
Change #1189604 merged by Dzahn:
[operations/puppet@production] zuul::executor: let executor connect to zookeeper on the host machine
If I am strict about the wording of this ticket then it's resolved.
executor is deployed.
whether it works together with the other components is to be tested though
I am claiming this as resolved.. an executor is deployed. And we have a separate task that will be about testing the whole setup once other missing parts have been created.
Since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275972 added a firewall rule the executors can now talk to zookeeper on the main machines.