Maniphest T252027

debian-installer: partman doesn't allow lvm LVs to be reused when reimaging
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Kormat
	May 6 2020, 3:30 PM

Description

The d-i partitioner, partman, wipes all lvm VGs/LVs as part of initialization. You can tell it not to, with d-i partman-lvm/device_remove_lvm boolean false, but then auto-partitioning doesn't continue. This makes it impossible to re-use an existing lvm LV when doing automated partitioning.

Details:

partman =>
initial_auto, which switches on partman-auto/method
'regular': =>
- autopartition =>
- auto_init_disks (see below)
'lvm': =>
- autopartition-lvm =>
- auto_lvm_prepare =>
- auto_init_disks (see below)
auto_init_disks =>
prepare_new_labels =>
device_remove_lvm, which.. does what it sounds like:
- Remove all LVs on the disk
- Removes the VG

Details

Subject	Repo	Branch	Lines +/-
install: Prevent full wipe of dbprov2003 data by changing its recipe	operations/puppet	production	+2 -2
install_server: Revert d-i-test to original partition scheme.	operations/puppet	production	+1 -23
install_server: Fix reuse-db.cfg recipe	operations/puppet	production	+3 -3
install_server: Fix issue with reuse-parts.cfg on multi-disk machines.	operations/puppet	production	+3 -0
install_server: Test reuse-parts on metal.	operations/puppet	production	+30 -20
install_server: Allow reuse of partitions during reimage.	operations/puppet	production	+274 -55

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Kormat	T252027 debian-installer: partman doesn't allow lvm LVs to be reused when reimaging
		Resolved		Kormat	T254982 reuse-parts.sh: provide feedback to user when something fails

Event Timeline

Kormat created this task.May 6 2020, 3:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2020, 3:30 PM

Kormat mentioned this in T251768: Make partman/custom/no-srv-format.cfg work.May 6 2020, 3:31 PM

colewhite triaged this task as Medium priority.May 6 2020, 4:48 PM

It is (probably) possible to work around this by not setting partman-auto/method and using partman/early_command to prepopulate the metadata that partman uses. It's not clear how much work is involved, though, beyond "some".

I think it would merit a PoC at least, so we can evaluate if it is 100% possible, so we can decide how to proceed further.
We need to involve the Infra Foundations team on this though cc @MoritzMuehlenhoff

After a lot more investigation, this is bordering on infeasible. It would require re-implementing a Lot of partman in order to make it work.

Some notes:

partman is a series of shell scripts, roughly 11.5k lines across 130+ files.
It spawns a parted server, and interacts it by doing rpc over 2 fifos.
The code undocumented, and full of unspecified interdependencies.
partman/early_command runs before any real setup is done. Trying to manually trigger the setup parts we need without invoking parts we don't want is very complex. You need a fairly complete understanding of how everything fits together to make this work.

Here's as far as i've managed to get:

d-i     partman/early_command   string \
        . /lib/partman/lib/base.sh; \
        . /lib/partman/lib/recipes.sh; \
        . /lib/partman/lib/auto-shared.sh; \
        set -ex; \
        mkdir -p /var/lib/partman; \
        for i in /lib/partman/init.d/*; do echo $i | grep -q 'early_command$' || $i; done; \
        touch /var/lib/partman/initial_auto; \
        perform_recipe $(dev_to_partman $(debconf-get partman-auto/disk)) "" es

It causes d-i to hang at "Starting up the partitioner".

Sad news :-/
If that is the case, I don't think there is much point on going forward with this, specially if we'd need to patch partman in many places - and then maintaining those patches.

Kormat mentioned this in T252195: In-place conversion from LVM to normal partition.May 8 2020, 12:26 PM

elukey added a subscriber: elukey.May 21 2020, 1:03 PM

elukey mentioned this in T234629: Move the Analytics infrastructure to Debian Buster.May 22 2020, 2:05 PM

I have a solution. A script that re-uses the existing / (formatted) and /srv (retained as-is) partitions by hooking into the partman internals.

P11300 reuse-parts.sh

1	#!/bin/sh
2
3	[ -e /tmp/reuse ] && exit 0;
4	touch /tmp/reuse
5
6	#export DEBIAN_FRONTEND=noninteractive
7	. /lib/partman/lib/base.sh
8	. /lib/partman/lib/recipes.sh
9
10	log() {
11	logger -t reuse "$@"
12	}
13
14	[ $(debconf-get partman-auto/method) = "reuse" ] \|\| { log "Skipping, partman-auto/method != reuse"; exit 0; }
15
16	disk=$(debconf-get partman-auto/disk)
17	log "Disk: $disk"
18	dev="$DEVICES/$(echo $disk \| sed 's:/:=:g')"
19	cd $dev \|\| { log "ERROR: $disk doesn't exist"; exit 1; }
20
21	open_dialog PARTITIONS
22	while { read_line num id size type fs path name; [ "$id" ]; }; do
23	case "$num" in
24	1)
25	log "====> Reuse / partition (with format)"
26	[ "$fs" = "ext4" ] \|\| { log "ERROR: expected fs ext4 for partition $num, got $fs instead"; break; }
27	echo format > $id/method
28	touch $id/format
29	touch $id/use_filesystem
30	echo "$fs" > $id/filesystem
31	mkdir -p $id/options
32	echo / > $id/mountpoint
33	touch $id/existing
34	touch $id/formatable
35	;;
36	esac
37	done
38	close_dialog
39
40	cd "$DEVICES"/=dev=mapper=* \|\| { log "ERROR: wrong number of lvm logical volumes found: $(echo "$DEVICES"/=dev=mapper=*)"; exit 1; }
41	log "====> Reuse /srv lvm partition (retain data)"
42	open_dialog PARTITIONS
43	read_line num id size type fs path name
44	[ "$fs" = "xfs" ] \|\| { log "ERROR: expected fs xfs for partition $num, got $fs instead"; break; }
45	echo keep > $id/method
46	touch $id/existing
47	echo "$fs" > $id/filesystem
48	echo /srv > $id/mountpoint
49	mkdir -p $id/options
50	touch $id/use_filesystem
51	close_dialog
52
53	log "Running update_all"
54	update_all
55	log "update_all result: $?"

Thanks for working on this! I'd need to read up on some of the Partman guts for a full review, but the general approach seems totally workable.

As for the question on how to include this script in the install environment; wget-ing it in partman/early_command is actually the sanest approach for now. With a PXE boot most of d-i gets downloaded via udebs from the Debian mirrors, so patching partman packages locally isn't an alternative here. We do rebuild the images currently, but this only adds firmware needed by some NICs, it doesn't rebuild debian-installer by itself.

That said, we're definitely not the only ones with the need to reimage a Debian server while retaining /srv. One other option (in addition to start with using partman/early_command) would be make the script a little more generic (like adding a profile option partman/retain_partition=/srv which enables your script into /lib/partman/display.d/ and submit it as a bug report and merge request on salsa.debian.org. When this lands into the next Debian release we can then simply switch to it and others can also use and enhance it.

Change 601761 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Allow reuse of partitions during reimage. [WIP]

https://gerrit.wikimedia.org/r/601761

gerritbot added a project: Patch-For-Review.Jun 2 2020, 3:15 PM

Change 601761 merged by Kormat:
[operations/puppet@production] install_server: Allow reuse of partitions during reimage.

https://gerrit.wikimedia.org/r/601761

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 10:10 AM

Change 603961 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Test reuse-parts on metal.

https://gerrit.wikimedia.org/r/603961

gerritbot added a project: Patch-For-Review.Jun 9 2020, 10:59 AM

Change 603961 merged by Kormat:
[operations/puppet@production] install_server: Test reuse-parts on metal.

https://gerrit.wikimedia.org/r/603961

Mentioned in SAL (#wikimedia-operations) [2020-06-09T12:22:10Z] <kormat> reimaging sretest1002 T252027

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['sretest1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006091223_kormat_33763.log.

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

Of which those FAILED:

['sretest1002.eqiad.wmnet']

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 1:11 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['sretest1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006091348_kormat_101863.log.

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

Of which those FAILED:

['sretest1002.eqiad.wmnet']

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['sretest1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006091351_kormat_104069.log.

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

and were ALL successful.

Change 604020 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Fix issue with reuse-parts.cfg on multi-disk machines.

https://gerrit.wikimedia.org/r/604020

gerritbot added a project: Patch-For-Review.Jun 9 2020, 2:18 PM

Change 604020 merged by Kormat:
[operations/puppet@production] install_server: Fix issue with reuse-parts.cfg on multi-disk machines.

https://gerrit.wikimedia.org/r/604020

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['sretest1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006091427_kormat_134282.log.

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 3:11 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1077.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006100751_kormat_18957.log.

Mentioned in SAL (#wikimedia-operations) [2020-06-10T07:52:41Z] <kormat> reimaging db1077 T252027

Completed auto-reimage of hosts:

['db1077.eqiad.wmnet']

Of which those FAILED:

['db1077.eqiad.wmnet']

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1077.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006100812_kormat_35212.log.

Completed auto-reimage of hosts:

['db1077.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1077.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006100856_kormat_69412.log.

Completed auto-reimage of hosts:

['db1077.eqiad.wmnet']

and were ALL successful.

Change 604315 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Fix reuse-db.cfg recipe

https://gerrit.wikimedia.org/r/604315

gerritbot added a project: Patch-For-Review.Jun 10 2020, 9:24 AM

Change 604315 merged by Kormat:
[operations/puppet@production] install_server: Fix reuse-db.cfg recipe

https://gerrit.wikimedia.org/r/604315

I'm pronouncing this Resolved \o/

I've successfully reimaged sretest1002 and db1007 using reuse-parts. We're now going to start using it more generally. I've opened T254982: reuse-parts.sh: provide feedback to user when something fails to follow-up on, but this task can now be closed.

Anyone who wants to use reuse-parts: i'm very happy to provide guidance/debugging if something fails, etc. Just poke me on irc, or file a task and assign it to me.

kormat (the partman guy now, apparently)

Marostegui awarded a token.Jun 10 2020, 9:49 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2020, 10:11 AM

Kormat closed subtask T254982: reuse-parts.sh: provide feedback to user when something fails as Resolved.Jun 11 2020, 9:18 AM

Change 604636 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Revert d-i-test to original partition scheme.

https://gerrit.wikimedia.org/r/604636

gerritbot added a project: Patch-For-Review.Jun 11 2020, 9:25 AM

Change 604636 merged by Kormat:
[operations/puppet@production] install_server: Revert d-i-test to original partition scheme.

https://gerrit.wikimedia.org/r/604636

Maintenance_bot removed a project: Patch-For-Review.Jun 11 2020, 1:10 PM

Change 618718 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install: Prevent full wipe of dbprov2003 data by changing its recipe

https://gerrit.wikimedia.org/r/618718

gerritbot added a project: Patch-For-Review.Aug 6 2020, 9:10 AM

Change 618718 merged by Jcrespo:
[operations/puppet@production] install: Prevent full wipe of dbprov2003 data by changing its recipe

https://gerrit.wikimedia.org/r/618718

debian-installer: partman doesn't allow lvm LVs to be reused when reimagingClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

debian-installer: partman doesn't allow lvm LVs to be reused when reimaging
Closed, ResolvedPublic
Actions

Related Objects
Search...