Page MenuHomePhabricator

puppet problems mounting cinder volumes (and suggested fixes)
Closed, ResolvedPublic

Description

I recently tried to use cinderutils::ensure / profile::labs::cindermount::srv in puppet to mount cinder volumes, as is mentioned in:

https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Cinder

as an alternative to manually running wmcs-prepare-cinder-volume.

When doing that I ran into some issues:

a) puppet runs the command but fails to mount the volume with the error message "Refusing to format volume in non-interactive mode without --allow-unattended-format""

I uploaded a suggested patch at gerrit:1056606 that would add that --allow-unattended-format parameter not being fully sure if that's really ok. Seems the most obvious fix attempt though.

b) When I try to use the "min_gb" and "max_gb" parameters to give the "size hints" [1] I can't set a size of 1GB. (my volume is 1GB). If I set min_gb to 1 that is considered "not larger than 1GB" and it fails to mount. If I skip min_gb it defaults to 10GB and therefore also won't mount. If I set it to 0.8, puppet fails because that isn't an integer.

gerrit:1057000 is a suggested patch for that that simply allows floating point numbers OR integers. The calculations made using it are in bytes, so _I think_ that would just work.

Could you please review or even merge my changes or leave comments either here or directly on Gerrit?

Thank you very much.

[1]

<snip> from cinderutils::ensure ...

# Unfortunately we don't have a way to distinguish between attached volumes
#  other than by size. That means that if this class is used to mount volumes
#  that were re-attached after use on a different host, the behavior is somewhat
#  undefined: any volume can get mounted at any of the requested mount points.
#  This can be worked around by mounting the volumes by hand (using prepare_cinder_volume
#  in interactive mode) or by providing size hints via min_gb and max_gb if you need
#  to e.g. mount a small volume at /small and a big volume at /big.

Event Timeline

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Change #1057000 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] cinderutils: allow floating point numbers for min_gb and max_gb

https://gerrit.wikimedia.org/r/1057000

Change #1056606 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] cinderutils: add --allow-unattended-format when preparing volumes

https://gerrit.wikimedia.org/r/1056606

Dzahn updated the task description. (Show Details)

This is relevant for me in the wikistats projected and sre-collab for T371066. So we could let puppet mount volumes without manual commands at all. Not a huge blocker but certainly would be nice.

Mentioned in SAL (#wikimedia-cloud-feed) [2024-08-01T19:11:32Z] <mutante> attaching volume 'gitlab-prod-logs' to instance 'gitlab-prod-1002' T371066 | running 'sudo wmcs-prepare-cinder-volume' manually and answering interactive questions to mount/format it (once T371573 is resolved will do this with puppet)

LSobanski triaged this task as Medium priority.Aug 6 2024, 7:04 AM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.

Change #1057000 merged by Dzahn:

[operations/puppet@production] cinderutils: allow floating point numbers for min_gb and max_gb

https://gerrit.wikimedia.org/r/1057000

Change #1060877 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] wikistats: re-add puppetized mounting of cinder volume

https://gerrit.wikimedia.org/r/1060877

Change #1060877 merged by Dzahn:

[operations/puppet@production] wikistats: re-add puppetized mounting of cinder volume

https://gerrit.wikimedia.org/r/1060877

Change #1056606 merged by Andrew Bogott:

[operations/puppet@production] cinderutils: add --allow-unattended-format when preparing volumes

https://gerrit.wikimedia.org/r/1056606

Thanks for the merge. let me test before closing it as resolved.

I tested this on a fresh instance when applying the puppet role for the first time.

The mount works now (though of course it still needs a manual action in Horizon to attach it to the new instance).

And when de-taching and then trying to run puppet again to re-attach you have to manually comment out the previous mount in /etc/fstab for it to work.

But regardless the issues I reported here are now fixed.

Thanks again!