At present we don't have infrastructure support for building aarch64 containers or packages. This leaves us with a couple of options for running dcl on aarch64:
(2) seems the easiest, if the performance is acceptable
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | jhathaway | T337970 Create tooling for disposable dev environments | |||
| Open | jhathaway | T349450 dcl: Version one feedback | |||
| Stalled | jhathaway | T360804 macOS aarch64 support |
@SLyngshede-WMF let's move discussion to phabricator. I tried the updated patch on my x86_64 hackintosh with minikube 1.32:
$ dcl lab start /usr/local/bin/minikube start --profile=eqiad --cpus=max --memory=5g --container-runtime=cri-o --dns-domain=k8s.lan --extra-config=kubeadm.pod-network-cidr=10.64.0.0/16 --service-cluster-ip-range=10.65.0.0/16 --cni=kindnet --driver=qemu2 --network=socket_vmnet 😄 [eqiad] minikube v1.32.0 on Darwin 13.4.1 ✨ Using the qemu2 driver based on user configuration 👍 Starting control plane node eqiad in cluster eqiad 🔥 Creating qemu2 VM (CPUs=16, Memory=5120MB, Disk=20000MB) ... 🎁 Preparing Kubernetes v1.28.3 on CRI-O 1.24.1 ... ▪ kubeadm.pod-network-cidr=10.64.0.0/16 ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring CNI (Container Networking Interface) ... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🔎 Verifying Kubernetes components... 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "eqiad" cluster and "default" namespace by default Waiting for coredns to come up..................... /usr/local/bin/minikube --profile=eqiad ssh sudo podman run --quiet --network kindnet --rm --privileged aptman/qus -- -r cat ./qemu-binfmt-conf.sh | sh -s -- --path=/qus/bin -r /usr/local/bin/minikube --profile=eqiad ssh sudo podman run --quiet --network kindnet --rm --privileged aptman/qus -s -- -p aarch64 cat ./qemu-binfmt-conf.sh | sh -s -- --path=/qus/bin -p aarch64 --suffix -static Setting /qus/bin/qemu-aarch64-static as binfmt interpreter for aarch64 <snip> $ minikube -p eqiad ssh minikube -p eqiad ssh $ sudo -i # podman run --quiet --network kindnet --rm -t arm64v8/fedora uname -m aarch64 #
With the sample dcl yaml config, I am able to run an aarch64 container on x86_64. I am curious whether the same works on a aarch64 host.
- Use qemu to run x86_64 containers on an aarch64 VM
FYI, this is going to be so pretty slow (anecdotal stories speak of standard puppet CI taking up to 15mins instead of a few secs, other workloads vary). See T320811 for the bigger picture, the rest of the tasks in ARM support spell out the various other problems met by developers.
Yeah, I was curious if it was too slow to be usable, the change was relatively simple, so I thought we could test it at least.
For somethings I believe it should be fine, the AMD64 "emulation" is pretty good. We did see it struggle on the Puppet CLI container, to the point of it being somewhat painful to use. It boils down to whether or not the thing it runs is somewhat predictable or not. Interpreters aren't fast, but anything that's compiled is mostly workable.
After doing some exploration I don't think emulation is viable due to the complexity and the performance, so blocking this task until T320811 is resolved. In the interim running dcl on WMCS looks to be the best alternative.