Two volumes not deleting/creating on deployment-prep
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TheresNoTime
	May 31 2022, 8:38 PM

Description

Volume 491066a4-16a9-4ce4-a9f6-182660616d77 has been "deleting" for over an hour, and volume 3c31b9df-e4c6-4a50-a9a1-f5c992e17747 has been "creating" for a little less than that — that doesn't feel right..?

Related Objects

Mentioned In: T309437: Create deployment-deploy04 as future secondary/upgrade

Event Timeline

TheresNoTime created this task.May 31 2022, 8:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2022, 8:38 PM

(wrong tag, apologies cloud-services-team)

In T309659#7972123, @TheresNoTime wrote:

(wrong tag, apologies cloud-services-team)

You had it right the first time. :) We have a Herald rule that normally would have changed cloud-services-team to cloud-services-team (Kanban).

RhinosF1 subscribed.May 31 2022, 9:40 PM

TheresNoTime mentioned this in T309437: Create deployment-deploy04 as future secondary/upgrade.Jun 1 2022, 4:26 PM

Mentioned in SAL (#wikimedia-cloud) [2022-06-01T16:37:32Z] <taavi> root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-182660616d77 for T309659

I suspect this one is caused by RabbitMQ losing the commands to create/delete those instances. I tried restating various Cinder services on cloudcontrols without any help. Fixed the first one with

root@cloudcontrol1005:~# openstack volume show 491066a4-16a9-4ce4-a9f6-182660616d77
+--------------------------------+--------------------------------------+
| Field                          | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2022-05-28T14:18:20.000000           |
| description                    |                                      |
| encrypted                      | False                                |
| id                             | 491066a4-16a9-4ce4-a9f6-182660616d77 |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | deploy04                             |
| os-vol-host-attr:host          | cloudcontrol1003@rbd#RBD             |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | deployment-prep                      |
| properties                     |                                      |
| replication_status             | None                                 |
| size                           | 40                                   |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | deleting                             |
| type                           | standard                             |
| updated_at                     | 2022-05-31T18:07:18.000000           |
| user_id                        | samtar                               |
+--------------------------------+--------------------------------------+
root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-182660616d77

and then deleting it via Horizon. Not sure if I can do the same thing for the second one or if that would break something.

Thank you @Majavah — I dare not create another one to connect to deploy04, seeing how well that went with 3c31b9df-e4c6-4a50-a9a1-f5c992e17747 😺

In T309659#7974447, @TheresNoTime wrote:

Thank you @Majavah — I dare not create another one to connect to deploy04, seeing how well that went with 3c31b9df-e4c6-4a50-a9a1-f5c992e17747 😺

I don't think there's any reason why you should not do that. The RabbitMQ issues I am referencing should be well resolved at this point.

That worked! (43071b78-084e-416f-9465-9e2b812c8310)

Volumes deleted

Two volumes not deleting/creating on deployment-prepClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Two volumes not deleting/creating on deployment-prep
Closed, ResolvedPublic
Actions