Page MenuHomePhabricator

cinder volumes stuck in detaching, deleting states
Closed, ResolvedPublic

Description

I seem to be unable to manage our volumes in the wikilink project. I'm trying to delete one volume wikilink-nfs and detatch and extend (or replace) another docker-data-root. After much fiddling around, including shutting the attached instance for docker-data-root, the volumes are still stuck. I finally gave up and started the instance back up and was able to remount docker-data-root and resume services for now, but we're very close to a full volume.

Event Timeline

Andrew subscribed.

Hello @jsn.sherman. There seem to be a few races/weak points in the code that manages volume attachment so sometimes things get stuck in an inconsistent state.

If you would specity your complete desired end state (which volumes deleted, which preserved and where attached) I can do some behind-the-scenes work to try to get things repaired.

Hello @jsn.sherman. There seem to be a few races/weak points in the code that manages volume attachment so sometimes things get stuck in an inconsistent state.

If you would specity your complete desired end state (which volumes deleted, which preserved and where attached) I can do some behind-the-scenes work to try to get things repaired.

Thank you! The desired end state is:

  • wikilink-nfs - deleted
  • docker-data-root expanded to 40GiB - attached to prod

I went ahead and shut the prod instance off, hoping that the volume docker-data-root may be detatched, expanded, attached without disruption or corruption.

What about the 'wikilink-backup' volume? Leave as-is?

What about the 'wikilink-backup' volume? Leave as-is?

yep, I was able to make changes to wikilink-backup last week, so it's good!

I deleted wikilink-nfs.

I detached and expanded docker-data-root. I did not reattach it because there are few fun bits left for you.

  • Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.
  • Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

I deleted wikilink-nfs.

I detached and expanded docker-data-root.

Thanks!

I did not reattach it because there are few fun bits left for you.

  • Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.

I'm happy to use another route. What is my other self-service option?

  • Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

I can definitely take care of that; I'm comfortable working with block devices and filesystems.

I deleted wikilink-nfs.

I detached and expanded docker-data-root.

Thanks!

I did not reattach it because there are few fun bits left for you.

  • Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.

I'm happy to use another route. What is my other self-service option?

I don't really have an alternative -- probably the best thing is to just try a few reboots and make sure that you get the same thing each time :) There's no behavioral difference between using horizon or the APIs directly.

  • Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

I can definitely take care of that; I'm comfortable working with block devices and filesystems.

This is looking good on initial boot. I'm going to let data collection catch up and then give it a reboot to verify that things are happy.

jsn.sherman moved this task from QA to Done on the Moderator-Tools-Team (Kanban) board.

looks good on reboot; thanks @Andrew!