Most of the canary instances that failed, failed with the following error:
dcaro@cloudcontrol1003:~$ for vm in $(sudo wmcs-openstack server list --project cloudvirt-canary --name 'canary' -f json | jq '.[] | select(.Status|test("ERROR")) | .ID' --raw-output); do sudo wmcs-openstack server show "$vm" -f json | jq '{name,fault}'; done ... { "name": "canary1022-01", "fault": { "code": 500, "created": "2021-02-22T11:13:24Z", "message": "Build of instance 4747a794-19fe-4536-ad81-3f6fee8a504e aborted: Image 6b67c8a1-6356-464d-a885-0576d7263e51 is unacceptable: Image has no associated data", "details": "Traceback (most recent call last):\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 4025, in clone_fallback_to_fetch\n backend.clone(context, disk_images['image_id'])\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py\", line 980, in clone\n reason=reason)\nnova.exception.ImageUnacceptable: Image 6b67c8a1-6356-464d-a885-0576d7263e51 is unacceptable: No image locations are accessible\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3/dist-packages/nova/compute/manager.py\", line 2437, in _build_and_run_instance\n block_device_info=block_device_info)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 3539, in spawn\n block_device_info=block_device_info)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 3925, in _create_image\n injection_info, fallback_from_host)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 4033, in _create_and_inject_local_root\n instance, size, fallback_from_host)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 9367, in _try_fetch_image_cache\n trusted_certs=instance.trusted_certs)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py\", line 275, in cache\n *args, **kwargs)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py\", line 940, in create_image\n prepare_template(target=base, *args, **kwargs)\n File \"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py\", line 328, in inner\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py\", line 271, in fetch_func_sync\n fetch_func(target=target, *args, **kwargs)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py\", line 4027, in clone_fallback_to_fetch\n libvirt_utils.fetch_image(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/nova/virt/libvirt/utils.py\", line 420, in fetch_image\n images.fetch_to_raw(context, image_id, target, trusted_certs)\n File \"/usr/lib/python3/dist-packages/nova/virt/images.py\", line 153, in fetch_to_raw\n fetch(context, image_href, path_tmp, trusted_certs)\n File \"/usr/lib/python3/dist-packages/nova/virt/images.py\", line 144, in fetch\n trusted_certs=trusted_certs)\n File \"/usr/lib/python3/dist-packages/nova/image/api.py\", line 199, in download\n trusted_certs=trusted_certs)\n File \"/usr/lib/python3/dist-packages/nova/image/glance.py\", line 346, in download\n reason='Image has no associated data')\nnova.exception.ImageUnacceptable: Image 6b67c8a1-6356-464d-a885-0576d7263e51 is unacceptable: Image has no associated data\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3/dist-packages/nova/compute/manager.py\", line 2161, in _do_build_and_run_instance\n filter_properties, request_spec)\n File \"/usr/lib/python3/dist-packages/nova/compute/manager.py\", line 2525, in _build_and_run_instance\n reason=e.format_message())\nnova.exception.BuildAbortException: Build of instance 4747a794-19fe-4536-ad81-3f6fee8a504e aborted: Image 6b67c8a1-6356-464d-a885-0576d7263e51 is unacceptable: Image has no associated data\n" } } ...