In T350586#9312988, @Andrew wrote:

Hello @jsn.sherman. There seem to be a few races/weak points in the code that manages volume attachment so sometimes things get stuck in an inconsistent state.

If you would specity your complete desired end state (which volumes deleted, which preserved and where attached) I can do some behind-the-scenes work to try to get things repaired.

Thank you! The desired end state is:

• wikilink-nfs - deleted
• docker-data-root expanded to 40GiB - attached to prod

I went ahead and shut the prod instance off, hoping that the volume docker-data-root may be detatched, expanded, attached without disruption or corruption.

What about the 'wikilink-backup' volume? Leave as-is?

In T350586#9313075, @Andrew wrote:

What about the 'wikilink-backup' volume? Leave as-is?

yep, I was able to make changes to wikilink-backup last week, so it's good!

I deleted wikilink-nfs.

I detached and expanded docker-data-root. I did not reattach it because there are few fun bits left for you.

• Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.

• Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

In T350586#9313136, @Andrew wrote:

I deleted wikilink-nfs.

I detached and expanded docker-data-root.

Thanks!

I did not reattach it because there are few fun bits left for you.

• Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.

I'm happy to use another route. What is my other self-service option?

• Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

I can definitely take care of that; I'm comfortable working with block devices and filesystems.

oh, I somehow missed that we had programmatic access!
https://wikitech.wikimedia.org/wiki/Help:Using_OpenStack_APIs

In T350586#9313151, @jsn.sherman wrote:

In T350586#9313136, @Andrew wrote:

I deleted wikilink-nfs.

I detached and expanded docker-data-root.

Thanks!

I did not reattach it because there are few fun bits left for you.

• Attaching. You can do this with the UI but be warned that the two volumes may get mounted in a less-than-deterministic way.

I'm happy to use another route. What is my other self-service option?

I don't really have an alternative -- probably the best thing is to just try a few reboots and make sure that you get the same thing each time :) There's no behavioral difference between using horizon or the APIs directly.

• Expanding the file-system. The volume itself is bigger now but you'll need to expand the actual filesystem. I can figure out how to do that if need be :)

I can definitely take care of that; I'm comfortable working with block devices and filesystems.

This is looking good on initial boot. I'm going to let data collection catch up and then give it a reboot to verify that things are happy.