In order to retain SBOM metadata in buildx based PipelineLib image builds, Docker Engine must be configured to use the containerd snapshotter. That option is only available in Docker >= 26.x. Debian trixie includes Docker Engine version 26.1.5, so let's upgrade integration agent hosts to trixie.
Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | dduvall | T440668 ST6.1.4 Refactor Pipelinelib to use buildx and generate SBOMs | |||
| Resolved | dduvall | T440670 Upgrade integration-agent-docker instances to trixie |
Event Timeline
Note the production agents (contint1003 and contint2003) are already on Trixie and thus with Docker 26.1.5 :)
Dan has created a new Trixie based instance:
| integration-agent-docker-1095 |
| https://horizon.wikimedia.org/project/instances/edfd9985-3e3f-4cdd-8a5e-9429ae475f90/ |
And it has been polled in Jenkins already π
@dduvall something I forgot, we have a runbook to build the integration instance, maybe you were aware of it and used it: https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup It might need some polishing, I am not sure how up to date it is.
From another thread, I have learned Ahmon fixed the step that required deleting the Puppet server cert (rm -fR /var/lib/puppet/ssl). That got addressed by Ahmon in T429413. Monte and Ahmon have build a new instance to serve as a proxy cache and had hit that, they should add profile::puppet::agent::puppetserver_ca_cert which would be the final step to address the provision issue on instances. You can sync up with Monte on it :)
This morning I had the job https://integration.wikimedia.org/ci/job/integration-config-zuul-layout-validate/ stalling with a timeout when running on that instance. The repro:
$ ssh integration-agent-docker-1095.integration.eqiad1.wikimedia.cloud $ mkdir src $ sudo docker run --rm -it --volume "$(pwd)"/src:/src --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm docker-registry.wikimedia.org/releng/zuul-cloner:0.3.0-s8 --color --verbose --map /zuul-clonemap.yaml --workspace /src --cache-dir /srv/git https://gerrit.wikimedia.org/r/p integration/config <stall>
The same work on another instance:
$ ssh integration-agent-docker-1085.integration.eqiad1.wikimedia.cloud $ mkdir src $ sudo docker run --rm -it --volume "$(pwd)"/src:/src --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm docker-registry.wikimedia.org/releng/zuul-cloner:0.3.0-s8 --color --verbose --map /zuul-clonemap.yaml --workspace /src --cache-dir /srv/git https://gerrit.wikimedia.org/r/p integration/config /opt/zuul/lib/python2.7/site-packages/zuul/lib/cloner.py:65: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. self.clone_map = yaml.load(clone_map_file).get('clonemap') INFO:zuul.Cloner:Loaded map containing 4 rules INFO:zuul.CloneMapper:Workspace path set to: /src DEBUG:zuul.CloneMapper:Using integration/config as destination (unmatched) INFO:zuul.CloneMapper:Mapping projects to workspace... INFO:zuul.CloneMapper: integration/config -> /src/integration/config ...
Thus there is something differing somewhere though I haven't investigated.
Meanwhile, I have disabled the agent in Jenkins.
Thanks, @hashar. It appears that zuul-cloner hangs while doing import git and that hangs during an execution of git version in Git.refresh().
root@37dfb3fa160f:/# /opt/zuul/bin/zuul-cloner --help
^CTraceback (most recent call last):
File "/opt/zuul/bin/zuul-cloner", line 5, in <module>
from zuul.cmd.cloner import main
File "/opt/zuul/lib/python2.7/site-packages/zuul/cmd/cloner.py", line 24, in <module>
import zuul.lib.cloner
File "/opt/zuul/lib/python2.7/site-packages/zuul/lib/cloner.py", line 16, in <module>
import git
File "/opt/zuul/lib/python2.7/site-packages/git/__init__.py", line 83, in <module>
refresh()
File "/opt/zuul/lib/python2.7/site-packages/git/__init__.py", line 73, in refresh
if not Git.refresh(path=path):
File "/opt/zuul/lib/python2.7/site-packages/git/cmd.py", line 227, in refresh
cls().version()
File "/opt/zuul/lib/python2.7/site-packages/git/cmd.py", line 548, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/opt/zuul/lib/python2.7/site-packages/git/cmd.py", line 1014, in _call_process
return self.execute(call, **exec_kwargs)
File "/opt/zuul/lib/python2.7/site-packages/git/cmd.py", line 735, in execute
**subprocess_kwargs
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1023, in _execute_child
data = _eintr_retry_call(os.read, errpipe_read, 1048576)
File "/usr/lib/python2.7/subprocess.py", line 125, in _eintr_retry_call
return func(*args)
KeyboardInterruptAfter some debugging, it appears to hit this bug in Python 2 related to subprocess.Popen(..., close_fds=True). A minimal reproduction inside the container is:
$ /opt/zuul/bin/python2.7 -c 'import subprocess; subprocess.Popen(["echo"], close_fds=True)'I'm pretty sure we work around the issue by configuring a lower max open files on the container, possibly host wide in /etc/docker/daemon.json. I will experiment with it today.
Thank you so much for having dig into it @dduvall , I confess I stopped at "something is broken", filed a task and did not plan to debugging :) It is quite fun to see the file limit has it us again it did cause slowness when generating the git info cache for MediaWiki and caused any scap sh commands to be delayed. Eventually tracked down back in 2019 T204762#5152768
As mentioned during our team meeting, it also had hit some workflow in GitLab T426827.
As baseline on Bookworm docker give us 1M limit:
$ ssh integration-agent-docker-1085.integration.eqiad1.wikimedia.cloud sudo docker run --rm -it --entrypoint=bash docker-registry.wikimedia.org/releng/zuul-cloner:0.3.0-s8 -c 'ulimit -n' 1048576
I'll have to remember to phase out that releng/zuul-cloner image. It is quite obsolete.
Change #1355127 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] ci: Set a default `nofile` ulimit for Docker containers
Mentioned in SAL (#wikimedia-releng) [2026-10-14T22:23:07Z] <dduvall> bringing integration-agent-docker-1095 back online after applying default-ulimit workaround (T440670)
Eventually got an alarm about disk filling up:
maintenance-disconnect-full-disks build 850945 integration-agent-docker-1095 (/: 97%, /srv: 6%, /var/lib/docker: 22%): still OFFLINE due to disk space
The /Β should not be that full cause the VM have two partitions dedicated to volatile data:
/srv: for the Jenkins workspace, caches etc
/var/lib/docker: for the Docker daemon (images, containers etc)
With the newer Docker, images get written to /var/lib/containerdΒ which is out of the /var/lib/docker partition and thus end up filing /. I found it with du -x / (-x skip files not on the same file system as the argument /):
$ sudo du --one-file-system -m -d3 /|sort -nr|head -n5 18370 / 16414 /var 15668 /var/lib 15417 /var/lib/containerd 1873 /usr
On a Bookworm instance, it is not used:
$ sudo du -m -s /var/lib/containerd/ 1 /var/lib/containerd/
Thus I guess we need another Docker daemon setting to have containerd to use /var/lib/docker partition? Or maybe nothing is written by Docker in /var/lib/docker anymore?
Twist: Docker reports different image sizes. For: docker-registry.wikimedia.org/releng/quibble-coverage:1.22.0-s3
- 3.15GB (Bookworm, integration-agent-docker-1085)
- 4.5GB (Trixie, integration-agent-docker-1095)
Change #1355127 merged by Dzahn:
[operations/puppet@production] ci: Set a default `nofile` ulimit for Docker containers
Mentioned in SAL (#wikimedia-operations) [2026-10-15T16:17:40Z] <mutante> contint1002/contint2002 - restart docker to apply a nofile ulimit for containers from gerrit:1355127 T440670
Mentioned in SAL (#wikimedia-operations) [2026-10-15T16:24:31Z] <mutante> contint1003/contint2003 - restart docker to apply a nofile ulimit for containers from gerrit:1355127 T440670
Mentioned in SAL (#wikimedia-releng) [2026-10-15T20:10:21Z] <dduvall> performing rolling restart of docker on integration-agent-docker instances to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1355127 (T440670)
Sorry for throwing more breakage your way, and thanks for catching and reporting the issue. I configured docker on that instance to use the containerd snapshotter (which is what eventually needs to happen on these instances so that metadata manifests created during buildx builds aren't lost prior to publishing).
Unfortunately it seems that even with containerd enabled, docker still uses /var/lib/docker for volumes, metadata, and logs. I think the easiest workaround is probably to configure containerd to store its data under the docker partition, so a directory such as /var/lib/docker/containerd. It feels a little strange given that containerd is a separate daemon and package, but considering docker is moving toward an embedded version of containerd and docker is the only thing that uses containerd on the hosts currently, I don't think it should cause any issues.
Change #1356455 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] ci: Ensure a containerd root beneath the docker data-root
Change #1356455 merged by Dzahn:
[operations/puppet@production] ci: Ensure a containerd root beneath the docker data-root
Mentioned in SAL (#wikimedia-cloud) [2026-10-29T16:50:06Z] <mutante> running puppet-git-sync-upstream manually on integration-puppetserver-01 to pick up 1356455 (containerd root beneath docker data-root) - T440670
The containerd config from 1356455 is live on the integration agents.
On integration-agent-docker-1070, /etc/containerd/config.toml now has the new path: root = "/var/lib/docker/containerd".
containerd still needs a restart on each agent to use it. Leaving that to Release-Engineering.
Mentioned in SAL (#wikimedia-releng) [2026-11-03T16:38:26Z] <dduvall> pruning all images and restarting containerd/docker on integration-agent-docker-1095 to deploy T440670
Mentioned in SAL (#wikimedia-releng) [2026-11-04T19:22:40Z] <dduvall> taking integration-agent-docker-107[0-5] offline for migration to trixie based agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T19:24:18Z] <dduvall> provisioning integration-agent-docker-109[6-9] trixie based agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T20:01:09Z] <dduvall> added new integration-agent-docker-{1097,1098,1099,1100,1101} trixie based agents to jenkins (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T20:03:29Z] <dduvall> removing integration-agent-docker-{1071,1072,1073,1074,1075} agents from jenkins and deleting instances (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T22:32:36Z] <dduvall> taking integration-agent-docker-{1076,1077,1078,1079,1080,1081} offline to be replaced by trixie agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T22:40:56Z] <dduvall> creating new integration-agent-docker-{1102,1103,1104,1105,1106,1107} trixie based instances/agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T22:57:14Z] <dduvall> registered new integration-agent-docker-{1102,1103,1104,1105,1106,1107} trixie based agents with jenkins (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T22:59:23Z] <dduvall> deleting integration-agent-docker-{1076,1077,1078,1079,1081,1082} agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:02:37Z] <dduvall> taking integration-agent-docker-{1083,1084,1085,1086,1087,1088} offline to be replaced by trixie agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:04:10Z] <dduvall> creating new integration-agent-docker-{1108,1109,1110,1111,1112,1113} trixie based instances/agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:27:07Z] <dduvall> registered new integration-agent-docker-{1108,1109,1110,1111,1112,1113} trixie based agents with jenkins (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:27:50Z] <dduvall> deleting integration-agent-docker-{1083,1084,1085,1086,1087,1088} agents and instances (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:31:39Z] <dduvall> taking integration-agent-docker-{1089,1090,1091,1092,1093} offline to be replaced by trixie agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:33:37Z] <dduvall> creating new integration-agent-docker-{1114,1115,1116,1117,1118} trixie based instances/agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:55:01Z] <dduvall> registered new integration-agent-docker-{1114,1115,1116,1117,1118} trixie based agents with jenkins (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-04T23:55:21Z] <dduvall> deleting integration-agent-docker-{1089,1090,1091,1092,1093} agents and instances (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T00:04:58Z] <dduvall> creating new integration-agent-puppet-docker-{1006,1007} trixie based instances/agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T00:19:17Z] <dduvall> taking integration-agent-puppet-docker-{1004,1005} offline to be replaced by trixie agents (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T00:21:11Z] <dduvall> deleting integration-agent-puppet-docker-{1004,1005} agents and instances (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T00:21:29Z] <dduvall> registered new integration-agent-puppet-docker-{1006,1007} trixie based agents with jenkins (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T00:27:41Z] <dduvall> all integration-agent-docker and integration-agent-puppet-docker agents have been migrated to trixie (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T17:43:54Z] <dduvall> creating new integration-agent-docker-9998 trixie based instances/agent to replace integration-agent-docker-9999 (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T20:58:29Z] <dduvall> deleting integration-agent-docker-9999 following replacement (T440670)
Reopening since I missed that the integration-agent-qemu instance has docker installed.
Mentioned in SAL (#wikimedia-releng) [2026-11-05T21:05:42Z] <dduvall> creating integration-agent-qemu-1005 trixie based agent to replace integration-agent-qemu-1004 (T440670)
@hashar just noting that I ran into issues with provisioning a new integration-agent-qemu instance-1005. The debian-11-ci.qcow2 image fails to build during the installation of packages due to the missing bullseye distribution on security.debian.org.
I worked around the issue for now by copying the existing image from integration-agent-qemu-1004, but we'll need to fix that long term. I'm assuming we'll want to refactor the ci-build-images.sh script in puppet to make a bookworm or trixie image.
Mentioned in SAL (#wikimedia-releng) [2026-11-05T21:35:29Z] <dduvall> registered integration-agent-qemu-1005 with jenkins. note the debian-11 image was copied from integration-agent-qemu-1004 due to build failures (T440670)
Mentioned in SAL (#wikimedia-releng) [2026-11-05T21:36:52Z] <dduvall> deleting integration-agent-qemu-1004 agent and instance following its replacement (T440670)
@dduvall yeah copying the qcow2 image is good enough. I don't think we should bother updating/upgrading that image.
For reference, Krinkle had set it a while ago in order to validate Fresh which uses Docker. The script in operations/puppet: https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/modules/profile/files/ci/ci-build-images.sh shows it is quite outdated:
url=https://cloud.debian.org/images/cloud/bullseye/20220121-894/debian-11-nocloud-amd64-20220121-894.qcow2
It is crippled with performance cause QEmu does not benefit from hardware acceleration. So maybe eventually I'll get podman rootless inside docker (if that is at all possible), or something lightweight :)
Thx for having migrated it!
Thanks, @hashar!
Resolved once again. Note the following remaining non-trixie instances. Migration of these instances is out of this task's scope.
| integration-cumin | bullseye |
| integration-puppetserver-01 | bookworm |
| integration-castor06 | bookworm |
| integration-agent-pkgbuilder-1006 | bookworm |
| integration-agent-pkgbuilder-1005 | bookworm |
Thx!
Cumin, there might be a task about it already.
For the QEMu image, I filed T443872 to track the upgrade.