Today I attempted to delete a cluster in the zuul project and it remains in a DELETE_FAILED status. I was unable to delete it via OpenTofu, the openstack CLI, or Horizon.
- project zuul ID c26d9d326bdf464fa1025939ded7e5a2
- cluster ID cf02bf6c-4a82-4ee6-812d-44d042e44887
$ openstack coe cluster show cf02bf6c-4a82-4ee6-812d-44d042e44887 +----------------------+----------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------+----------------------------------------------------------------------------------------------------------------------------------+ | status | DELETE_FAILED | | health_status | UNKNOWN | | cluster_template_id | f5ec807c-f594-49e4-9031-d517790aaff9 | | node_addresses | ['172.16.17.91', '172.16.19.34', '172.16.16.60', '172.16.19.45', '172.16.19.243', '172.16.16.39', '172.16.18.242', | | | '172.16.16.32', '172.16.17.221', '172.16.18.31'] | | uuid | cf02bf6c-4a82-4ee6-812d-44d042e44887 | | stack_id | 0c3c525b-33e8-4448-834b-a3a393fed72c | | status_reason | ERROR: Internal Error | | created_at | 2025-07-17T23:32:14+00:00 | | updated_at | 2026-06-05T18:50:40+00:00 | | coe_version | v1.28.9-rancher1 | | labels | {'cloud_provider_tag': 'v1.27.3', 'keystone_auth_enabled': 'true', 'hyperkube_prefix': 'docker.io/rancher/', | | | 'auto_scaling_enabled': 'false', 'cinder_csi_enabled': 'true', 'monitoring_enabled': 'true', 'k8s_keystone_auth_tag': 'v1.27.3', | | | 'octavia_ingress_controller_tag': 'v1.27.3', 'magnum_auto_healer_tag': 'v1.27.3', 'container_runtime': 'containerd', | | | 'cloud_provider_enabled': 'true', 'calico_tag': 'v3.26.4', 'containerd_version': '1.6.31', 'kube_tag': 'v1.28.9-rancher1', | | | 'auto_healing_controller': 'magnum-auto-healer', 'auto_healing_enabled': 'false', 'master_lb_floating_ip_enabled': 'false', | | | 'cinder_csi_plugin_tag': 'v1.27.3', 'containerd_tarball_sha256': | | | '75afb9b9674ff509ae670ef3ab944ffcdece8ea9f7d92c42307693efa7b6109d'} | | labels_overridden | {} | | labels_skipped | {} | | labels_added | {} | | fixed_network | VXLAN/IPv6-dualstack | | fixed_subnet | vxlan-dualstack-ipv4 | | floating_ip_enabled | False | | faults | {} | | keypair | magnum-keypair | | api_address | https://172.16.16.192:6443 | | master_addresses | ['172.16.16.192'] | | master_lb_enabled | False | | create_timeout | None | | node_count | 10 | | discovery_url | https://etcd-discovery.wmcloud.org/c4b3ec29e33f1fdded194f381976459d | | docker_volume_size | 10 | | master_count | 1 | | container_version | 1.12.6 | | name | zuul-k8s-v128 | | master_flavor_id | g4.cores2.ram4.disk20 | | flavor_id | g4.cores8.ram32.disk20 | | health_status_reason | {'api': 'The cluster zuul-k8s-v128 is not accessible.'} | | project_id | c26d9d326bdf464fa1025939ded7e5a2 | +----------------------+----------------------------------------------------------------------------------------------------------------------------------+
Inspecting the stack doesn't seem to give any clues either:
$ openstack stack resource list --long 0c3c525b-33e8-4448-834b-a3a393fed72c +---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+ | secgroup_rule_udp_kube_mi | 9a270a5c-3dde-4b2a-a000- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | nion_pods_cidr | 17074de0cd47 | pRule | | | | | secgroup_rule_tcp_kube_mi | 70aac9d9-fd4a-4a9e-8fe5- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | nion | 93e5a098c886 | pRule | | | | | secgroup_rule_tcp_kube_mi | b5079d59-0ea9-4b9b-afae- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | nion_pods_cidr | 059c0f1e44f0 | pRule | | | | | secgroup_rule_udp_kube_mi | fefcf25f-b289-4c4e-9c83- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | nion | c1d273f8a32d | pRule | | | | | kube_cluster_deploy | 99658124-6e2b-4ee5-baab- | OS::Heat::SoftwareDeploym | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 781eced016be | ent | | | | | kube_cluster_config | 28520971-0b2a-47cb-b57c- | OS::Heat::SoftwareConfig | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 182ea04f37d5 | | | | | | kube_minions | 4f11f82c-ca2c-4fd1-a7fa- | OS::Heat::ResourceGroup | UPDATE_COMPLETE | 2025-07-29T22:35:08Z | zuul-k8s-v128-qnv6fefra4wd | | | d3fc428cae0b | | | | | | secgroup_kube_minion | 93262c28-2eb4-4d4c-8225- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 0a0679a9d2f9 | p | | | | | worker_nodes_server_group | ceeecf5c-edd7-4ec8-8b13- | OS::Nova::ServerGroup | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 3ffcd58eb425 | | | | | | etcd_address_lb_switch | 23b8d173-f55d-4981-aa11- | Magnum::ApiGatewaySwitche | UPDATE_COMPLETE | 2025-07-29T22:35:07Z | zuul-k8s-v128-qnv6fefra4wd | | | eaa9d88dc673 | r | | | | | api_address_floating_swit | 32cd9daf-9b0d-4454-8a6d- | Magnum::FloatingIPAddress | UPDATE_COMPLETE | 2025-07-29T22:35:08Z | zuul-k8s-v128-qnv6fefra4wd | | ch | cc41c6aff27e | Switcher | | | | | api_address_lb_switch | 43229fc6-e59d-4e25-a252- | Magnum::ApiGatewaySwitche | UPDATE_COMPLETE | 2025-07-29T22:35:07Z | zuul-k8s-v128-qnv6fefra4wd | | | 4b49989d748e | r | | | | | kube_masters | 10e5f159-4bb5-4410-abf3- | OS::Heat::ResourceGroup | UPDATE_COMPLETE | 2025-07-29T22:34:58Z | zuul-k8s-v128-qnv6fefra4wd | | | fc4c3f274f53 | | | | | | master_nodes_server_group | 81140f84-d2a8-476d-8eff- | OS::Nova::ServerGroup | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 764eb85201a1 | | | | | | etcd_lb | 7c3f8a47-fb04-4c74-b4a4- | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:56Z | zuul-k8s-v128-qnv6fefra4wd | | | 6eb211fb89f5 | ist-packages/magnum/drive | | | | | | | rs/common/templates/lb_et | | | | | | | cd.yaml | | | | | secgroup_kube_master | c2965c22-56e5-4d0e-864d- | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd | | | 0bf4f7529201 | p | | | | | api_lb | 6b4e21af-a50e-4441-8bbd- | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:56Z | zuul-k8s-v128-qnv6fefra4wd | | | 26f291df21cd | ist-packages/magnum/drive | | | | | | | rs/common/templates/lb_ap | | | | | | | i.yaml | | | | | network | 07852a3c-3d21-459a-a77b- | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:52Z | zuul-k8s-v128-qnv6fefra4wd | | | f20e41b12338 | ist-packages/magnum/drive | | | | | | | rs/common/templates/netwo | | | | | | | rk.yaml | | | | +---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+
Thanks in advance for the help!