Page MenuHomePhabricator

Unable to delete volume group accounts-oauth
Closed, ResolvedPublic

Description

I would like to delete the volume group accounts-oauth (3802a64c-81a2-4226-91a3-c10f1811b522) and the two volumes within it - oauth-www (daceff9e-51be-420f-8acb-35b9b02d1fa5) and oauth-db (99351df5-e183-4970-bdaf-5a0c8956f6c3).


If I try to use Actions -> Remove Volumes from Group, I get this error:

Error: Errors occurred in removing volumes from group. Details

Invalid filters group_id are found in query options. (HTTP 400) (Request-ID: req-c2feaf5f-cc6d-4e3b-a68f-d3f82d3d517f)

If I instead use Actions -> Manage Volumes, and remove one or more volumes from the group, I get the following message:

Success: Updated volumes for group.

... followed by a series of these messages, approximately one every 15 seconds:

Screenshot from 2023-07-20 18-10-13.png (692×1 px, 126 KB)

All error messages are the same (though with a different Request-ID each time):

Error: Unable to retrieve group details

Invalid filters group_id are found in query options. (HTTP 400) (Request-ID: req-5ceeca67-1b3b-440c-99ce-302f76cb917c)

Following this, the volume group gets stuck in an "Updating" status, and the flood of errors continues. I actually had this issue a few weeks ago, decided to look at something else for a while and forgot about the problem. Upon logging in to Horizon again yesterday, I was still getting these messages whenever I'm on the volume groups page.

This error can be stopped by updating the volume group description via Actions -> Edit Group; doing so returns the status to "Available" and stops the errors occurring.

I also get the "Invalid filters group_id" error if I try and click the hyperlinked name of the volume group.


I appear to be completely unable to get rid of this volume group, and thus unable to get rid of the two unneeded cinder volumes in this group.

Event Timeline

I'm not sure that this is all of the problem, but /some/ of the problem is that you have volumes based off of snapshots based off of the volumes that are in that volume group. To delete things we'd need to start at the end of the chain and delete everything. If you delete accounts-oauth-g-database (and possibly accounts-oauth-b-database) then things might go a bit better. Would you like me to do that, or do you need those?

Hmmm... maybe this has come from a fundamental misunderstanding of how snapshots and volumes work on OpenStack. I had thought that a volume created from a snapshot was then independent of that snapshot, allowing the snapshot to be then removed leaving the volume? (FWIW, this is exactly how EBS volumes and EBS snapshots work on AWS).

I do indeed need the other volumes; I'll do some switcharoo later and try to untangle that mess.

Looks like I've managed to get accounts-oauth-g-database stuck in detaching mode; would you mind helping that one along?

Looks like I've managed to get accounts-oauth-g-database stuck in detaching mode; would you mind helping that one along?

I think this is detached now, finally.

Thanks Andrew!

I've recreated the volume accounts-oauth-g-database (and instance accounts-oauth-g) to not be based on snapshots.

The volume accounts-oauth-b-database is now not required, and can be deleted. It doesn't look like I can delete it though, as it's got the snapshot accounts-oauth-database-20230704b. When I try to delete that snapshot, I get a "scheduled for deletion" message, but the "Messages" tab in Horizon is showing delete snapshot:Snapshot is busy..

I'll let this sit for a day or two and see if OpenStack frees stuff up in the background before trying again.

OK, so now what I'm seeing is:

  • 0/4 volume snapshot quota usage, which feels suspicious
  • 2 volume snapshots, which don't want to delete with a delete snapshot:Snapshot is busy. error (in the messages tab)
  • 2 volumes (accounts-oauth-b-database and oauth-www) which I can't delete with You are not allowed to delete volume: accounts-oauth-b-database errors (as a popup)
  • and the volume group.

I think I'm going to need admin help to continue dealing with this :(

As such, feel free to delete any/all of the following objects:

  • volume accounts-oauth-b-database
  • volume oauth-www
  • volume oauth-db
  • snapshot accounts-oauth-database-20230704b
  • snapshot oauth-db-20221228b
  • volgroup accounts-oauth

Sorry again for the chaos I seem to have unwittingly caused here!

Cinder is being a real jerk about this! I've had to make some manual db changes but I think I have almost everything deleted. The only loose end is the volume group which seems to be forever in the 'deleting' state.

My long-term advice here is: probably don't use volume groups!

I finally gave up on the API and manually marked this deleted in the database.