Page MenuHomePhabricator

kafka-jumbo.cfg partman recipe creation/troubleshooting
Closed, ResolvedPublic8 Estimated Story Points

Description

{placeholder task, rob will populate shortly}
This task will track the specific issue of writing a new (and working) partman recipe for these hosts.

I've currently pushed live the following version: P5941
This creates the following:

root@kafka-jumbo1001:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs            13G  8.7M   13G   1% /run
/dev/sda1        37G  1.2G   34G   4% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
root@kafka-jumbo1001:~# fdisk -l

Disk /dev/sda: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa0ddde00

Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *        2048   78125055   78123008  37.3G 83 Linux
/dev/sda2       78127102 1952446463 1874319362 893.8G  5 Extended
/dev/sda5       78127104 1952446463 1874319360 893.8G 8e Linux LVM

Disk /dev/sdb: 21.8 TiB, 24001350991872 bytes, 46877638656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 49659450-F008-4F5A-925A-C7B5B2979993

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 46877636607 46877634560 21.8T Linux LVM

Disk /dev/mapper/kafka--jumbo1001--vg-data: 22.7 TiB, 24960995164160 bytes, 48751943680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

So it is creating the LVM on both disks, even though the recipe uses device{ /dev/sda }, which seems like it should work. Additionally, it is only creating an empty LVM on sdb. (I confirmed this by entering manual partition mode, wiping the lvm, and rerunning the auto recipe and it creates the empty LVM on an empty sdb.) It seems odd that it creates the full / and LVM (but no /srv mount) in sda, and only an LVM in sdb.

Event Timeline

So I realized that:
d-i partman-auto/choose_recipe es

was in the recipe, and isn't needed since it doesn't exist in this recipe. I've removed it. Additionally, I've added in the format and method format lines. The newly tested (livehacked not puppetized) version of the file:

# configuration to create:
#  * hardware raid on kafka-jumbo hosts
#  * sda raid1: 2 * 1TB mounted as /
#  * sdb raid10: 12 * 4TB in 90% lvm as /srv
#  ** 10% empty space for later/emergency growth


d-i     partman-auto/method     string  lvm
d-i     partman-auto/disk       string  /dev/sda /dev/sdb

# the install makes sure we want to wipe the lvm
d-i     partman-lvm/device_remove_lvm   boolean true
d-i     partman-lvm/confirm     boolean true
d-i     partman-lvm/confirm_nooverwrite boolean true
d-i     partman/confirm boolean true

d-i     partman-auto/expert_recipe      string  \
                40000 40000 12000000 ext4               \
                        $primary{ }             \
                        $bootable{ }            \
                        method{ format }        \
                        format{ }               \
                        device{ /dev/sda }      \
                        use_filesystem{ }       \
                        filesystem{ ext4 }      \
                        mountpoint{ / }         \
                .                               \
                64 1000 10000000 ext4           \
                        lv_name{ data }         \
                        $defaultignore{ }       \
                        $lvmok{ }               \
                        method{ format }        \
                        format{ }               \
                        mountpoint{ /srv }      \
                        device{ /dev/sdb }      \
                        use_filesystem{ }       \
                        filesystem{ ext4 }      \
                        options/nobarrier { nobarrier } \
                        options/noatime { noatime } \
                .

d-i     partman-auto-lvm/guided_size    string  90%

d-i     partman/choose_partition        \
                select  finish
d-i     partman-partitioning/confirm_write_new_label    boolean true

d-i     partman/confirm_nooverwrite     boolean true
d-i     partman-basicfilesystems partman-basicfilesystems/no_swap boolean false

Running install to see if it changes anything, like perhaps formatting and mounting the sdb /srv. It did prompt for the swap not being there, and I realize that not having it with d-i made it not register, and with it, it likely needs to be set to true.

The result is promising!

root@kafka-jumbo1001:~# fdisk -l

Disk /dev/sda: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe5d5475c

Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *        2048   78125055   78123008  37.3G 83 Linux
/dev/sda2       78127102 1952446463 1874319362 893.8G  5 Extended
/dev/sda5       78127104 1952446463 1874319360 893.8G 8e Linux LVM

Disk /dev/sdb: 21.8 TiB, 24001350991872 bytes, 46877638656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AB8275AA-F443-41FC-8181-2285300CAD36

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 46877636607 46877634560 21.8T Linux LVM

Disk /dev/mapper/kafka--jumbo1001--vg-data: 22.7 TiB, 24960995164160 bytes, 48751943680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@kafka-jumbo1001:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8232596,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=13176716k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=84)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/mapper/kafka--jumbo1001--vg-data on /srv type ext4 (rw,relatime,data=ordered)
root@kafka-jumbo1001:~# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                    10M     0   10M   0% /dev
tmpfs                                   13G  8.7M   13G   1% /run
/dev/sda1                               37G  1.2G   34G   4% /
tmpfs                                   32G     0   32G   0% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
tmpfs                                   32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/kafka--jumbo1001--vg-data   23T   20K   22T   1% /srv

It still shouldn't be setting up any kind of LVM on the SDA, so not sure why it is.

Ok, so putting the recipe info to ignore noswap requires:
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false
putting in any kind if d-i makes partman ignore that line somehow. Also true means it prompts back to the partitinion menu due to no swap (tested.)

# configuration to create:
#  * hardware raid on kafka-jumbo hosts
#  * sda raid1: 2 * 1TB mounted as /
#  * sdb raid10: 12 * 4TB in 90% lvm as /srv
#  ** 10% empty space for later/emergency growth


d-i     partman-auto/method     string  lvm
d-i     partman-auto/disk       string  /dev/sda /dev/sdb

# the install makes sure we want to wipe the lvm
d-i     partman-lvm/device_remove_lvm   boolean true
d-i     partman-lvm/confirm     boolean true
d-i     partman-lvm/confirm_nooverwrite boolean true
d-i     partman/confirm boolean true

d-i     partman-auto/expert_recipe      string  \
                40000 40000 12000000 ext4               \
                        $primary{ }             \
                        $bootable{ }            \
                        method{ format }        \
                        format{ }               \
                        device{ /dev/sda }      \
                        use_filesystem{ }       \
                        filesystem{ ext4 }      \
                        mountpoint{ / }         \
                .                               \
                64 1000 10000000 ext4           \
                        lv_name{ data }         \
                        $defaultignore{ }       \
                        $lvmok{ }               \
                        method{ format }        \
                        format{ }               \
                        mountpoint{ /srv }      \
                        device{ /dev/sdb }      \
                        use_filesystem{ }       \
                        filesystem{ ext4 }      \
                        options/nobarrier { nobarrier } \
                        options/noatime { noatime } \
                .

d-i     partman-auto-lvm/guided_size    string  90%

d-i     partman/choose_partition        \
                select  finish
d-i     partman-partitioning/confirm_write_new_label    boolean true

d-i     partman/confirm_nooverwrite     boolean true
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false

Results in:

root@kafka-jumbo1001:~# fdisk -l

Disk /dev/sda: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xef85a9b8

Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *        2048   78125055   78123008  37.3G 83 Linux
/dev/sda2       78127102 1952446463 1874319362 893.8G  5 Extended
/dev/sda5       78127104 1952446463 1874319360 893.8G 8e Linux LVM

Disk /dev/sdb: 21.8 TiB, 24001350991872 bytes, 46877638656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F5B83DE0-9EC6-4231-BBC1-8682BCCD76E5

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 46877636607 46877634560 21.8T Linux LVM

Disk /dev/mapper/kafka--jumbo1001--vg-data: 22.7 TiB, 24960995164160 bytes, 48751943680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@kafka-jumbo1001:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8232595,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=13176716k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=30778)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/mapper/kafka--jumbo1001--vg-data on /srv type ext4 (rw,relatime,data=ordered)
root@kafka-jumbo1001:~# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                    10M     0   10M   0% /dev
tmpfs                                   13G  8.7M   13G   1% /run
/dev/sda1                               37G  1.2G   34G   4% /
tmpfs                                   32G     0   32G   0% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
tmpfs                                   32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/kafka--jumbo1001--vg-data   23T   20K   22T   1% /srv
root@kafka-jumbo1001:~#

So this recipe is the closest yet (sets up sdb with /srv, doesn't prompt about overwriting or no swap) but puts in invalid lvm data/partitions in sda.

Change 374645 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] further tweaks to kafka-jumbo

https://gerrit.wikimedia.org/r/374645

First thing that came up in my mind is a previous fight with Partman that ended up in this note:

https://wikitech.wikimedia.org/wiki/PartMan#Important_note_about_RAID

But this may not be the case, /sda and /sdb partitions seems completely different from your fdisk output..

The following recipe is almost working fine, except that I didn't manage to create the srv logical volume yet for some reason:

# configuration:
#  * hardware raid on kafka-jumbo hosts
#  * sda raid1: 2 * 1TB mounted as /
#  * sdb raid10: 12 * 4TB in 90% lvm as /srv
#  ** 10% empty space for later/emergency growth
#
# layout:
# * Two LVM phisical volumes, vg-flex (sda) and vg-data (sdb)
# * /boot partition on /dev/sda1, no LVM
# * / partition on a vg-flex logical volume, called root
# * /srv partition on a vg-data logical volume, called srv

d-i	partman-auto/method	string	lvm
d-i	partman-auto/disk	string	/dev/sda /dev/sdb

# the install makes sure we want to wipe the lvm
d-i	partman-lvm/device_remove_lvm	boolean	true
d-i	partman-lvm/confirm	boolean	true
d-i	partman-lvm/confirm_nooverwrite	boolean	true
d-i	partman/confirm	boolean	true

d-i	partman-auto/choose_recipe	lvm

d-i     partman-auto/expert_recipe      string  \
		lvm ::				\
		300 300 300 ext4			\
			$primary{ }		\
			$bootable{ }		\
			method{ format }	\
			format{ }		\
			use_filesystem{ }	\
			filesystem{ ext4 }	\
			mountpoint{ /boot }	\
			device{ /dev/sda }	\
		.				\
		100000 300000 1000000000 ext4		\
			$defaultignore{ }		\
			$primary{ }		\
			method{ lvm }	\
			device{ /dev/sda }	\
			vg_name{ vg-flex }	\
		.				\
		40000 40000 40000 ext4			\
			$defaultignore{ }		\
			$primary{ }		\
			method{ keep }	\
			device{ /dev/sda }	\
		.				\
		500000 300 -1 ext4			\
			$defaultignore{ }		\
			$primary{ }		\
			method{ lvm }	\
			device{ /dev/sdb }	\
			vg_name{ vg-data }	\
		.				\
		40000 40000 40000 ext4		\
			$lvmok{ }		\
			method{ format }	\
			format{ }		\
			use_filesystem{ }	\
			filesystem{ ext4 }	\
			mountpoint{ / } 	\
			in_vg{ vg-flex } 	\
			lv_name{ root } 	\
		.				\
		10000 80000 100000000 ext4		\
			$lvmok{ }		\
			method{ format }	\
			format{ }		\
			use_filesystem{ }	\
			filesystem{ ext4 }	\
			in_vg{ vg-flex } 	\
			lv_name{ placeholder } 	\
		.
                40000 80000 21000000 ext4       \
                        $lvmok{ }               \
                        method{ format }        \
                        format{ }               \
                        use_filesystem{ }       \
                        filesystem{ ext4 }      \
                        mountpoint{ /srv }      \
                        in_vg{ vg-data }         \
                        lv_name{ srv }		\
                .

d-i	partman-auto-lvm/guided_size	string	90%

d-i	partman/choose_partition	\
		select	finish
d-i	partman-partitioning/confirm_write_new_label	boolean	true

d-i	partman/confirm_nooverwrite	boolean	true
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false

# do not prompt for 'no filesystem on partition'
d-i	partman-basicmethods/method_only	boolean false
d-i	partman-basicfilesystems/no_mount_point boolean false

The other attempts were a failure since on sda partman tried to always create LVM partitions, and any attempt to force it not to do it ended up in a failure to partition (like no root partition found).

Forgot a "\" after a ".", the where is left to the reader as exercise :D

This is what we end up with:

root@kafka-jumbo1001:~# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda3  vg-flex lvm2 a--  893.46g    0
  /dev/sdb1  vg-data lvm2 a--   21.83t    0
root@kafka-jumbo1001:~# lvs
  LV          VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  srv         vg-data -wi-ao----  21.83t
  placeholder vg-flex -wi-a----- 856.21g
  root        vg-flex -wi-ao----  37.25g
root@kafka-jumbo1001:~# fdisk -l

Disk /dev/sda: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdacb6560

Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *        2048     585727     583680   285M 83 Linux
/dev/sda2         585728   78710783   78125056  37.3G 83 Linux
/dev/sda3       78710784 1952446463 1873735680 893.5G 8e Linux LVM

Disk /dev/sdb: 21.8 TiB, 24001350991872 bytes, 46877638656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2369DC41-3CD7-405F-9727-462C1E3EEEA3

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 46877636607 46877634560 21.8T Linux LVM

Disk /dev/mapper/vg--flex-root: 37.3 GiB, 39996882944 bytes, 78118912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg--data-srv: 21.8 TiB, 24001346797568 bytes, 46877630464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg--flex-placeholder: 856.2 GiB, 919353688064 bytes, 1795612672 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

root@kafka-jumbo1001:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8232596,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=13176716k,mode=755)
/dev/mapper/vg--flex-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=18530)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/mapper/vg--data-srv on /srv type ext4 (rw,relatime,data=ordered)

root@kafka-jumbo1001:~# df -h
Filesystem                Size  Used Avail Use% Mounted on
udev                       10M     0   10M   0% /dev
tmpfs                      13G  8.7M   13G   1% /run
/dev/dm-0                  37G  1.2G   34G   4% /
tmpfs                      32G     0   32G   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                      32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                 268M   58M  192M  24% /boot
/dev/mapper/vg--data-srv   22T   20K   21T   1% /srv

So afaics it seems a good compromise, the only thing that puzzles me is why the root partition is on /dev/dm-0, but it may be something not important. (?)

The /boot partition might need more space, I used the prometheus.cfg configuration as baseline and forgot to update that one (2GB might be fine).

Weird! /dev/dm-0 is a LVM partition?

I can definitely see the following that looks correct:

root@kafka-jumbo1001:/# mount | grep root
/dev/mapper/vg--flex-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)

root@kafka-jumbo1001:/# lvs
  LV          VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  srv         vg-data -wi-ao----  21.83t
  placeholder vg-flex -wi-a----- 856.21g
  root        vg-flex -wi-ao----  37.25g

Change 375002 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Tune the kafka-jumbo.cfg partman recipe

https://gerrit.wikimedia.org/r/375002

The last code review removes the need for the 'placeholder' logical volume and removes a unused/not-necessary partition (probably a copy/pasta issue):

		40000 40000 40000 ext4			\
			$defaultignore{ }		\
			$primary{ }		\
			method{ keep }	\
			device{ /dev/sda }	\
		.

This is the result:

root@kafka-jumbo1001:~# fdisk -l

Disk /dev/sda: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7a592bc9

Device     Boot   Start        End    Sectors   Size Id Type
/dev/sda1  *       2048    9764863    9762816   4.7G 83 Linux
/dev/sda2       9764864 1952446463 1942681600 926.4G 8e Linux LVM

Disk /dev/sdb: 21.8 TiB, 24001350991872 bytes, 46877638656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D00C7251-7616-4350-A6FB-1670CA951562

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 46877636607 46877634560 21.8T Linux LVM

Disk /dev/mapper/vg--flex-root: 926.3 GiB, 994649833472 bytes, 1942675456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/vg--data-srv: 21.8 TiB, 24001346797568 bytes, 46877630464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

root@kafka-jumbo1001:~# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  vg-flex lvm2 a--  926.34g    0
  /dev/sdb1  vg-data lvm2 a--   21.83t    0

root@kafka-jumbo1001:~# lvs
  LV   VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  srv  vg-data -wi-ao----  21.83t
  root vg-flex -wi-ao---- 926.34g

root@kafka-jumbo1001:~# lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb                 8:16   0  21.8T  0 disk
└─sdb1              8:17   0  21.8T  0 part
  └─vg--data-srv  254:1    0  21.8T  0 lvm  /srv
sda                 8:0    0   931G  0 disk
├─sda2              8:2    0 926.4G  0 part
│ └─vg--flex-root 254:0    0 926.3G  0 lvm  /
└─sda1              8:1    0   4.7G  0 part /boot

I re-discovered a comment that I made for a previous partman recipe, namely:

#  ** The rest will be allocated to the 'unused' LV without mounting any
#  ** partition on it. This volume is a placeholder to overcome a partman
#     "feature" that forces the last LV defined to get the whole space left
#     in the PV independently from the size requested.

So this is probably a limitation of partman, and it looks exactly like the problem that I am seeing. I tried different values in the recipe for maximum size/priorities but nothing has changed. The other problem is that I can't make the 90% maximum usage for the logical volume as partman-auto-lvm/guided_size string 90% should guarantee (at least IIUC).

@RobH any suggestion about where the issue might be?

Nuria set the point value for this task to 13.
Nuria changed the point value for this task from 13 to 8.

Assuming that the 90% maximum limit of guided_size is not applied in the expert recipe (to be verified), I created the following:

root@kafka-jumbo1001:~# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  vg-flex lvm2 a--  926.34g    0
  /dev/sdb1  vg-data lvm2 a--   21.83t    0
root@kafka-jumbo1001:~# lvs
  LV               VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  srv              vg-data -wi-ao----  19.10t
  srv-placeholder  vg-data -wi-a-----   2.73t
  root             vg-flex -wi-ao---- 833.21g
  root-placeholder vg-flex -wi-a-----  93.13g
root@kafka-jumbo1001:~# df -h
Filesystem                Size  Used Avail Use% Mounted on
udev                       10M     0   10M   0% /dev
tmpfs                      13G  8.7M   13G   1% /run
/dev/dm-0                 821G  1.2G  778G   1% /
tmpfs                      32G     0   32G   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                      32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                 4.5G   66M  4.2G   2% /boot
/dev/mapper/vg--data-srv   20T   20K   19T   1% /srv

That seems good enough for the moment in my opinion :)

Change 375002 merged by Elukey:
[operations/puppet@production] Tune the kafka-jumbo.cfg partman recipe

https://gerrit.wikimedia.org/r/375002

Change 374645 abandoned by RobH:
further tweaks to kafka-jumbo

https://gerrit.wikimedia.org/r/374645