Once done, also edit Administration pages for Hadoop and Kafka that have scripts to create new nodes and partitions to make sure fstab is populated with UUID.
Description
Event Timeline
Did some Kafka reboots today and ran into the issue where /dev numbers are rearranged after reboot. I had to manually use UUIDs in fstab and reboot again to fix. I then proceeded to edit fstab and add UUIDs for every other broker as I rebooted it. The ones I did first still need their fstabs edited.
I did:
blkid, then copy/paste UUIDs for into fstab appropriately. Then umount /var/spool/kafka/* and mount -a, then look at mount -v | grep kafka | sort to see if things look right.
kafka1013, kafka1018, kafka1020 and kafka1022 have had fstabs edited with UUIDs. kafka1012 and kafka1014 still need to be done.
Quick script to match UUID and mountpoint:
elukey@kafka1012:~$ cat /proc/mounts | grep spool | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); echo -e "$uuid\t$mountpoint" ; done UUID=296c194d-2724-4ac1-a355-70e63d1580d7 /var/spool/kafka/a UUID=25239f66-549f-4bee-9c25-adf9dc8d5ff4 /var/spool/kafka/b UUID=efb2fbd4-37da-434f-82a5-4b78350b8466 /var/spool/kafka/c UUID=b7de45ba-25b0-49de-80bf-e5484ed10542 /var/spool/kafka/d UUID=47f6fe0c-2cf6-4df3-932d-9881571feaae /var/spool/kafka/e UUID=24a0117e-e24b-488c-8786-11b39d36cca6 /var/spool/kafka/f UUID=c65f08a6-5bda-42b8-9436-0a15ca138426 /var/spool/kafka/g UUID=139db99e-886d-41e3-bac4-bdefc4fdab7b /var/spool/kafka/h UUID=2ddaa0b6-715f-4f17-9340-4b2f94c996b7 /var/spool/kafka/i UUID=457cdef8-ece1-42f5-984e-367b45224153 /var/spool/kafka/j UUID=377e7539-ac72-4d46-be61-77c9d6e3aacd /var/spool/kafka/k UUID=23313468-69e2-4e55-a1e2-0cb472e97cd2 /var/spool/kafka/l
Validated on kafka1018:
elukey@kafka1018:~$ comm -12 <(cat /proc/mounts | grep spool | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); echo -e "$uuid\t$mountpoint" ; done | sort) <(cat /etc/fstab | grep spool |sort| awk '{print $1"\t"$2}') UUID=0e4d589d-be06-4782-a292-a36de30117c6 /var/spool/kafka/k UUID=1ea3a865-c906-4551-ab1e-70ae0ae27155 /var/spool/kafka/i UUID=46bfcaa4-5746-4809-8075-209a20ec26d8 /var/spool/kafka/c UUID=5cf6b088-d57e-48c8-a65a-a78cced66f6e /var/spool/kafka/a UUID=67a74799-0422-49b1-b79b-fd73ce0d1186 /var/spool/kafka/b UUID=6b11de43-0a83-425e-9c22-eddc46530e71 /var/spool/kafka/l UUID=7badbc40-0af6-491a-a366-495c19b6dd6c /var/spool/kafka/h UUID=b19afeae-6304-4547-aeeb-5bae57cc7a71 /var/spool/kafka/j UUID=c63e2368-6896-4113-a315-83e6a5f7bb64 /var/spool/kafka/g UUID=d301b1a1-c355-4b6c-bbf4-fd1a510ab598 /var/spool/kafka/f UUID=ddd21113-abfa-4481-847d-489d1402edbb /var/spool/kafka/d UUID=e8c9e786-3c16-4430-94ff-73ff2c217ec2 /var/spool/kafka/e
cat /proc/mounts | grep spool | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); letter=$(echo $mountpoint | cut -d "/" -f 5); echo -e "# Kafka log partition $letter\n$uuid\t$mountpoint\text4\tdefaults,noatime,data=writeback,nobh,delalloc\t0\t2" ; done
This one emits the fstab chunk of text to insert, like:
# Kafka log partition a UUID=296c194d-2724-4ac1-a355-70e63d1580d7 /var/spool/kafka/a ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition b UUID=25239f66-549f-4bee-9c25-adf9dc8d5ff4 /var/spool/kafka/b ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition c UUID=efb2fbd4-37da-434f-82a5-4b78350b8466 /var/spool/kafka/c ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition d UUID=b7de45ba-25b0-49de-80bf-e5484ed10542 /var/spool/kafka/d ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition e UUID=47f6fe0c-2cf6-4df3-932d-9881571feaae /var/spool/kafka/e ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition f UUID=24a0117e-e24b-488c-8786-11b39d36cca6 /var/spool/kafka/f ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition g UUID=c65f08a6-5bda-42b8-9436-0a15ca138426 /var/spool/kafka/g ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition h UUID=139db99e-886d-41e3-bac4-bdefc4fdab7b /var/spool/kafka/h ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition i UUID=2ddaa0b6-715f-4f17-9340-4b2f94c996b7 /var/spool/kafka/i ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition j UUID=457cdef8-ece1-42f5-984e-367b45224153 /var/spool/kafka/j ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition k UUID=377e7539-ac72-4d46-be61-77c9d6e3aacd /var/spool/kafka/k ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2 # Kafka log partition l UUID=23313468-69e2-4e55-a1e2-0cb472e97cd2 /var/spool/kafka/l ext4 defaults,noatime,data=writeback,nobh,delalloc 0 2
Mentioned in SAL (#wikimedia-operations) [2016-11-09T13:35:29Z] <elukey> stopping kafka* daemons on kafka1012 to upgrade its fstab with UUID (T147879)
kafka1012 is up and running. To sanity check the kafka spools mount point I did the following:
- for el in $(ls); do sudo find $el; done > /home/elukey/spools.txt BEFORE changing fstab
- for el in $(ls); do sudo find $el; done > /home/elukey/spools2.txt right after changing fstab and umount/mount -a.
- for el in $(ls); do sudo find $el; done > /home/elukey/spools3.txt after the reboot.
diff shows no change!
Mentioned in SAL (#wikimedia-operations) [2016-11-09T13:58:38Z] <elukey> stopping kafka* daemons on kafka1014 to upgrade its fstab with UUID (T147879)
Next steps:
- fix the fstabs of the hadoop nodes
- update the documentation for new hosts to use UUIDs.
Script adapted to an Hadoop node:
#!/bin/bash set -e # Root partition cat /proc/mounts | grep "mapper" | grep "root" | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); echo -e "#Root partition\n$uuid\t$mountpoint\text4\terrors=remount-ro\t0\t1" ; done # Journalnode partition cat /proc/mounts | grep "journalnode" | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); echo -e "#Hadoop JournalNode partition\n$uuid\t$mountpoint\text4\tdefaults,noatime\t0\t2" ; done # Swap partition cat /etc/fstab | grep "mapper" | grep "swap" | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); echo -e "#Swap\n$uuid\tnone\tswap\tsw\t0\t0" ; done # Hadoop partitions cat /proc/mounts | grep "/var/lib/hadoop/data" | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut -d " " -f 2| sed -e 's/"//g'); letter=$(echo $mountpoint | cut -d "/" -f 6); echo -e "# Kafka log partition $letter\n$uuid\t$mountpoint\text4\tdefaults,noatime\t0\t2" ; done
Mentioned in SAL (#wikimedia-operations) [2017-01-31T13:58:59Z] <elukey> rebooted analytics1039 to pick up uuids in fstab - T147879
Results look good!
elukey@analytics1039:~$ cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=32953052k,nr_inodes=8238263,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=6592852k,mode=755 0 0 /dev/mapper/analytics1039--vg-root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 none /sys/fs/pstore pstore rw,relatime 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0 /dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0 /dev/sdb1 /var/lib/hadoop/data/b ext4 rw,noatime,data=ordered 0 0 /dev/sdk1 /var/lib/hadoop/data/k ext4 rw,noatime,data=ordered 0 0 /dev/sdl1 /var/lib/hadoop/data/l ext4 rw,noatime,data=ordered 0 0 /dev/sdm1 /var/lib/hadoop/data/m ext4 rw,noatime,data=ordered 0 0 /dev/sdc1 /var/lib/hadoop/data/c ext4 rw,noatime,data=ordered 0 0 /dev/sdd1 /var/lib/hadoop/data/d ext4 rw,noatime,data=ordered 0 0 /dev/sde1 /var/lib/hadoop/data/e ext4 rw,noatime,data=ordered 0 0 /dev/sdf1 /var/lib/hadoop/data/f ext4 rw,noatime,data=ordered 0 0 /dev/sdg1 /var/lib/hadoop/data/g ext4 rw,noatime,data=ordered 0 0 /dev/sdh1 /var/lib/hadoop/data/h ext4 rw,noatime,data=ordered 0 0 /dev/sdi1 /var/lib/hadoop/data/i ext4 rw,noatime,data=ordered 0 0 /dev/sdj1 /var/lib/hadoop/data/j ext4 rw,noatime,data=ordered 0 0 /dev/mapper/analytics1039--vg-journalnode /var/lib/hadoop/journal ext4 rw,noatime,data=ordered 0 0 systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
Fixed some little mistakes and added documentation to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Worker_Nodes_.28DataNode_.26_NodeManager.29
We are going to reimage all the analytics hosts to Debian, so this task can be closed now.