Page MenuHomePhabricator

Magnum cluster stuck in DELETE_FAILED status
Closed, ResolvedPublic

Description

Today I attempted to delete a cluster in the zuul project and it remains in a DELETE_FAILED status. I was unable to delete it via OpenTofu, the openstack CLI, or Horizon.

  • project zuul ID c26d9d326bdf464fa1025939ded7e5a2
  • cluster ID cf02bf6c-4a82-4ee6-812d-44d042e44887
$ openstack coe cluster show cf02bf6c-4a82-4ee6-812d-44d042e44887
+----------------------+----------------------------------------------------------------------------------------------------------------------------------+
| Field                | Value                                                                                                                            |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------+
| status               | DELETE_FAILED                                                                                                                    |
| health_status        | UNKNOWN                                                                                                                          |
| cluster_template_id  | f5ec807c-f594-49e4-9031-d517790aaff9                                                                                             |
| node_addresses       | ['172.16.17.91', '172.16.19.34', '172.16.16.60', '172.16.19.45', '172.16.19.243', '172.16.16.39', '172.16.18.242',               |
|                      | '172.16.16.32', '172.16.17.221', '172.16.18.31']                                                                                 |
| uuid                 | cf02bf6c-4a82-4ee6-812d-44d042e44887                                                                                             |
| stack_id             | 0c3c525b-33e8-4448-834b-a3a393fed72c                                                                                             |
| status_reason        | ERROR: Internal Error                                                                                                            |
| created_at           | 2025-07-17T23:32:14+00:00                                                                                                        |
| updated_at           | 2026-06-05T18:50:40+00:00                                                                                                        |
| coe_version          | v1.28.9-rancher1                                                                                                                 |
| labels               | {'cloud_provider_tag': 'v1.27.3', 'keystone_auth_enabled': 'true', 'hyperkube_prefix': 'docker.io/rancher/',                     |
|                      | 'auto_scaling_enabled': 'false', 'cinder_csi_enabled': 'true', 'monitoring_enabled': 'true', 'k8s_keystone_auth_tag': 'v1.27.3', |
|                      | 'octavia_ingress_controller_tag': 'v1.27.3', 'magnum_auto_healer_tag': 'v1.27.3', 'container_runtime': 'containerd',             |
|                      | 'cloud_provider_enabled': 'true', 'calico_tag': 'v3.26.4', 'containerd_version': '1.6.31', 'kube_tag': 'v1.28.9-rancher1',       |
|                      | 'auto_healing_controller': 'magnum-auto-healer', 'auto_healing_enabled': 'false', 'master_lb_floating_ip_enabled': 'false',      |
|                      | 'cinder_csi_plugin_tag': 'v1.27.3', 'containerd_tarball_sha256':                                                                 |
|                      | '75afb9b9674ff509ae670ef3ab944ffcdece8ea9f7d92c42307693efa7b6109d'}                                                              |
| labels_overridden    | {}                                                                                                                               |
| labels_skipped       | {}                                                                                                                               |
| labels_added         | {}                                                                                                                               |
| fixed_network        | VXLAN/IPv6-dualstack                                                                                                             |
| fixed_subnet         | vxlan-dualstack-ipv4                                                                                                             |
| floating_ip_enabled  | False                                                                                                                            |
| faults               | {}                                                                                                                               |
| keypair              | magnum-keypair                                                                                                                   |
| api_address          | https://172.16.16.192:6443                                                                                                       |
| master_addresses     | ['172.16.16.192']                                                                                                                |
| master_lb_enabled    | False                                                                                                                            |
| create_timeout       | None                                                                                                                             |
| node_count           | 10                                                                                                                               |
| discovery_url        | https://etcd-discovery.wmcloud.org/c4b3ec29e33f1fdded194f381976459d                                                              |
| docker_volume_size   | 10                                                                                                                               |
| master_count         | 1                                                                                                                                |
| container_version    | 1.12.6                                                                                                                           |
| name                 | zuul-k8s-v128                                                                                                                    |
| master_flavor_id     | g4.cores2.ram4.disk20                                                                                                            |
| flavor_id            | g4.cores8.ram32.disk20                                                                                                           |
| health_status_reason | {'api': 'The cluster zuul-k8s-v128 is not accessible.'}                                                                          |
| project_id           | c26d9d326bdf464fa1025939ded7e5a2                                                                                                 |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------+

Inspecting the stack doesn't seem to give any clues either:

$ openstack stack resource list --long 0c3c525b-33e8-4448-834b-a3a393fed72c
+---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+
| resource_name             | physical_resource_id      | resource_type             | resource_status | updated_time         | stack_name                 |
+---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+
| secgroup_rule_udp_kube_mi | 9a270a5c-3dde-4b2a-a000-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
| nion_pods_cidr            | 17074de0cd47              | pRule                     |                 |                      |                            |
| secgroup_rule_tcp_kube_mi | 70aac9d9-fd4a-4a9e-8fe5-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
| nion                      | 93e5a098c886              | pRule                     |                 |                      |                            |
| secgroup_rule_tcp_kube_mi | b5079d59-0ea9-4b9b-afae-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
| nion_pods_cidr            | 059c0f1e44f0              | pRule                     |                 |                      |                            |
| secgroup_rule_udp_kube_mi | fefcf25f-b289-4c4e-9c83-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
| nion                      | c1d273f8a32d              | pRule                     |                 |                      |                            |
| kube_cluster_deploy       | 99658124-6e2b-4ee5-baab-  | OS::Heat::SoftwareDeploym | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 781eced016be              | ent                       |                 |                      |                            |
| kube_cluster_config       | 28520971-0b2a-47cb-b57c-  | OS::Heat::SoftwareConfig  | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 182ea04f37d5              |                           |                 |                      |                            |
| kube_minions              | 4f11f82c-ca2c-4fd1-a7fa-  | OS::Heat::ResourceGroup   | UPDATE_COMPLETE | 2025-07-29T22:35:08Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | d3fc428cae0b              |                           |                 |                      |                            |
| secgroup_kube_minion      | 93262c28-2eb4-4d4c-8225-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 0a0679a9d2f9              | p                         |                 |                      |                            |
| worker_nodes_server_group | ceeecf5c-edd7-4ec8-8b13-  | OS::Nova::ServerGroup     | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 3ffcd58eb425              |                           |                 |                      |                            |
| etcd_address_lb_switch    | 23b8d173-f55d-4981-aa11-  | Magnum::ApiGatewaySwitche | UPDATE_COMPLETE | 2025-07-29T22:35:07Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | eaa9d88dc673              | r                         |                 |                      |                            |
| api_address_floating_swit | 32cd9daf-9b0d-4454-8a6d-  | Magnum::FloatingIPAddress | UPDATE_COMPLETE | 2025-07-29T22:35:08Z | zuul-k8s-v128-qnv6fefra4wd |
| ch                        | cc41c6aff27e              | Switcher                  |                 |                      |                            |
| api_address_lb_switch     | 43229fc6-e59d-4e25-a252-  | Magnum::ApiGatewaySwitche | UPDATE_COMPLETE | 2025-07-29T22:35:07Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 4b49989d748e              | r                         |                 |                      |                            |
| kube_masters              | 10e5f159-4bb5-4410-abf3-  | OS::Heat::ResourceGroup   | UPDATE_COMPLETE | 2025-07-29T22:34:58Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | fc4c3f274f53              |                           |                 |                      |                            |
| master_nodes_server_group | 81140f84-d2a8-476d-8eff-  | OS::Nova::ServerGroup     | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 764eb85201a1              |                           |                 |                      |                            |
| etcd_lb                   | 7c3f8a47-fb04-4c74-b4a4-  | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:56Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 6eb211fb89f5              | ist-packages/magnum/drive |                 |                      |                            |
|                           |                           | rs/common/templates/lb_et |                 |                      |                            |
|                           |                           | cd.yaml                   |                 |                      |                            |
| secgroup_kube_master      | c2965c22-56e5-4d0e-864d-  | OS::Neutron::SecurityGrou | CREATE_COMPLETE | 2025-07-17T23:32:21Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 0bf4f7529201              | p                         |                 |                      |                            |
| api_lb                    | 6b4e21af-a50e-4441-8bbd-  | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:56Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | 26f291df21cd              | ist-packages/magnum/drive |                 |                      |                            |
|                           |                           | rs/common/templates/lb_ap |                 |                      |                            |
|                           |                           | i.yaml                    |                 |                      |                            |
| network                   | 07852a3c-3d21-459a-a77b-  | file:///usr/lib/python3/d | UPDATE_COMPLETE | 2025-07-29T22:34:52Z | zuul-k8s-v128-qnv6fefra4wd |
|                           | f20e41b12338              | ist-packages/magnum/drive |                 |                      |                            |
|                           |                           | rs/common/templates/netwo |                 |                      |                            |
|                           |                           | rk.yaml                   |                 |                      |                            |
+---------------------------+---------------------------+---------------------------+-----------------+----------------------+----------------------------+

Thanks in advance for the help!

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-cloud) [2026-06-07T17:40:22Z] <andrewbogott> wmcs.openstack.restart_openstack --cluster-name eqiad1 --all as step one in troubleshooting T428312

This is producing an internal permissions error, as though heat isn't allowed to delete the things it just created.

Restarting services didn't help, so the next step is to hook the policy engine and see who and what is not allowed. That might be a bit deep for a Sunday. @dduvall please let me know if this is blocking you and I will just god-mode-delete things and save my curiosity for another day.

Keystone says:

raise exception.Forbidden(_('Trustee has no delegated roles.'))

trust_role_id: e2dfeddb1c514120b467835266f4bc06 (k8s_admin)
current_effective_trustor_roles: ['38676f30eaeb44518bf7e144a73c8da6', 'f75a3c410bca4e96a1cf6ac103b0ccaf', '2cd63d467f754404bf3746fe63ee0698'] (admin, member, reader)

That fits, the trustor is bd808 and those are his roles. Did bd808 previously have the k8s_admin role and no longer has it, or is k8s_admin a service role used by heat and then misplaced somehow?

Anyway... I added these roles to bd808+zuul: k8s_admin and k8s_developer. After that I tried 'coe cluster delete' and it worked just fine. I then re-removed those roles from bd808 for the sake of neatness.

It's reasonable, as documented, that those roles would be needed for a user manipulating k8s clusters. What I don't understand is... how did anything like that ever work before? I would say that I've just been coasting on my adminship and not needing those roles for myself, except bd808 is /also/ a global admin.

...and after all that, I'm curious to hear if either of you (@dduvall or @bd808) can still create clusters. Or delete them, for that matter. My only current theory is that the 'edge case' we ran into was one of you creating and the other deleting although nothing in the code or policies indicates that that would matter.

Thanks, @Andrew! I was able to delete the cluster template via openstack coe cluster template delete just now FWIW. I'll attempt to re-create the template and cluster via https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning today and let you know how it goes.

dduvall claimed this task.

I tested a full create/destroy cycle via tofu and it worked. Thanks, @Andrew

Closing this for now although I'm still uncomfortable that I don't when we do and don't need those k8s_admin and k8s_developer roles