Page MenuHomePhabricator

Investigate better way of deferring activation of Labs LVM volumes (and corresponding snapshots) until after system boot
Closed, DeclinedPublic

Description

Today we were bitten by the fact that currently ALL LVM logical volumes on the labstore hosts get activated early on in the boot process, before the system is up and accessible. More specifically, in this case activation of all LVs took a very long time (approximately an hour) due to LVM scanning the volumes for COW tables for snapshots. In the mean time, there was no feedback about this process happening during the bootup, and during the system is only accessible over serial console. This process is not interruptible, so that means that the only way to interact with the system then is a power cycle.

We've worked around this for now by enabling the 'activationskip' flag on all Labs LVs and corresponding snapshots , which prevents any of these LVs from being activated unless manually overridden or started in the start-nfs script. However, this may not be the best way of going about it, as this will activate ALL LVs in the VG while there may be unrelated but legitimate reasons to prevent certain LVs from being activated.

We could look at an alternative way, e.g. by modifying settings in /etc/lvm/lvm.conf or explicitly activating specific Labs LVs and snapshots individually in start-nfs.

This should also involve testing the LVM snapshots load performance.

Event Timeline

mark created this task.Dec 16 2015, 11:13 AM
mark raised the priority of this task from to High.
mark updated the task description. (Show Details)
mark added a subscriber: mark.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2015, 11:13 AM
chasemp updated the task description. (Show Details)Dec 18 2015, 7:36 PM

Need to figure out if lvm snapshots need to be activated for COW to work

greg added a subscriber: greg.Sep 29 2016, 7:41 PM

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

chasemp closed this task as Declined.Feb 15 2017, 3:08 PM
chasemp added a subscriber: chasemp.

closing in favor of T158196