Page MenuHomePhabricator

Reimage the Debian Jessie Analytics worker nodes to Stretch.
Closed, ResolvedPublic8 Story Points

Description

The analytics1028->1069 worker nodes are still running Debian Jessie, meanwhile we have been successfully running analytics 1070->79 with Stretch without any issue. Managing two os at the same time could be cumbersome due to different package versions etc.., plus sooner or later we'll be asked to do so from the SRE team :)

This is not part of any goal up to now (April 2018), so we'll manage this task as best effort (so no deadlines set in stone about when it will be finished, the priority goes to all the goals).

High level procedure:

https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Worker_Reimage_(12_disk,_2_flex_bay_drives_-_analytics1028-analytics1057)

More details:

  1. Schedule downtime and drain Hadoop jobs
# On einsteinium
sudo icinga-downtime -d 7200 -r "prep for reimage" -h analytics1068 

# On the host
sudo puppet agent --disable "$USER - drain & reimage"
sudo systemctl stop hadoop-yarn-nodemanager
watch 'ps aux | grep yarn | grep -c -v grep'
# ...until no jvms/containers running
sudo systemctl stop hadoop-hdfs-datanode

Be EXTREMELY careful with analytics10[28,35,52], since they are the journal nodes!
Ensure that all three of them are up and running before reimage, otherwise the namenode will shutdown

  1. launch wmf-reimage
# On Neodymium
sudo -i wmf-auto-reimage -p T192557 analytics1068.eqiad.wmnet

When the script reboots the host, it will start another puppet run, that usually doesn't complete since it fails due to the /var/lib/hadoop dirs not present. Don't close the script, skip to next point.

  1. On the host, check with fdisk that all the disks have Linux fs partitions (a simple fdisk -l is fine).
  1. On the host:
sudo blkid | grep primary | awk '{print $2" "$1}' | sed -e 's/[:"]//g' | while read uuid partition;
do 
    letter=$(echo $partition| awk -F 'sd|1' '{print $2}');
    echo -e "# Hadoop datanode $letter partition\n$uuid\t/var/lib/hadoop/data/${letter}\text4\tdefaults,noatime\t0\t2";
done >> /etc/fstab

root@analytics1068:/# grep -c hadoop /etc/fstab
12

for el in b  c  d  e  f  g  h  i  j  k  l  m; do mkdir -p /var/lib/hadoop/data/${el}; done

mount -a
  1. If the hdfs and yarn users and groups were assigned differrent uid/gids, we need to chown all the hadoop lib files to the correct user:group.

Check to see if files are owned properly:

ls -ld /var/lib/hadoop/data/b/{yarn,hdfs}
drwx------ 3 hdfs hdfs 4096 Jul 31  2015 /var/lib/hadoop/data/b/hdfs
drwxr-xr-x 4 yarn yarn 4096 Jul 31  2015 /var/lib/hadoop/data/b/yarn

If these are each owned as 'hdfs:hdfs' and 'yarn:yarn', you can skip to step 6 now. Otherwise:

puppet agent --disable "$USER - chown step"

for letter in $(ls /var/lib/hadoop/data); do
  sudo chown -Rv yarn:yarn /var/lib/hadoop/data/${letter}/yarn &
  sudo chown -Rv hdfs:hdfs /var/lib/hadoop/data/${letter}/hdfs &
done
  1. Enable puppet and run, verifying that all the daemons are up and running as expected. The wmf-auto-reimage script will complete and report success in this task.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fdans moved this task from Incoming to Operational Excellence on the Analytics board.

Change 427702 merged by Elukey:
[operations/puppet@production] Set Debian Stretch as target OS for all the Hadoop worker nodes

https://gerrit.wikimedia.org/r/427702

Mentioned in SAL (#wikimedia-operations) [2018-04-20T11:27:47Z] <elukey> reimage analytics1068 to Debian Stretch - T192557

elukey updated the task description. (Show Details)Apr 20 2018, 11:32 AM
elukey updated the task description. (Show Details)

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1068.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804201138_elukey_25433.log.

elukey updated the task description. (Show Details)Apr 20 2018, 12:06 PM
elukey updated the task description. (Show Details)

Completed auto-reimage of hosts:

['analytics1068.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2018-04-23T13:55:35Z] <elukey> reimage analytics1067 to Debian Stretch - T192557

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1067.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804231358_elukey_17086.log.

Completed auto-reimage of hosts:

['analytics1067.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1065.eqiad.wmnet', 'analytics1066.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804241125_elukey_24097.log.

Completed auto-reimage of hosts:

['analytics1065.eqiad.wmnet', 'analytics1066.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1063.eqiad.wmnet', 'analytics1064.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804241223_elukey_7980.log.

Completed auto-reimage of hosts:

['analytics1064.eqiad.wmnet', 'analytics1063.eqiad.wmnet']

and were ALL successful.

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 25 2018, 6:03 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1061.eqiad.wmnet', 'analytics1062.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804251006_elukey_15551.log.

Completed auto-reimage of hosts:

['analytics1062.eqiad.wmnet', 'analytics1061.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1060.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804251353_otto_13303.log.

Ottomata updated the task description. (Show Details)Apr 25 2018, 2:22 PM

Completed auto-reimage of hosts:

['analytics1060.eqiad.wmnet']

and were ALL successful.

Ottomata updated the task description. (Show Details)Apr 25 2018, 6:54 PM

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1058.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804251907_otto_20455.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1059.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804251907_otto_20464.log.

Completed auto-reimage of hosts:

['analytics1058.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1059.eqiad.wmnet']

and were ALL successful.

Ottomata updated the task description. (Show Details)Apr 26 2018, 1:45 PM

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1057.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804261358_otto_25487.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1056.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804261358_otto_25463.log.

Completed auto-reimage of hosts:

['analytics1056.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1057.eqiad.wmnet']

and were ALL successful.

Ottomata updated the task description. (Show Details)Apr 26 2018, 3:44 PM

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1054.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804261602_otto_22371.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1055.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804261602_otto_22904.log.

Completed auto-reimage of hosts:

['analytics1055.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1054.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1051.eqiad.wmnet', 'analytics1053.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804270919_elukey_23876.log.

Completed auto-reimage of hosts:

['analytics1053.eqiad.wmnet', 'analytics1051.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1049.eqiad.wmnet', 'analytics1050.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804301323_elukey_31794.log.

Completed auto-reimage of hosts:

['analytics1049.eqiad.wmnet', 'analytics1050.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1047.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011604_otto_21206.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1048.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011604_otto_21224.log.

Ottomata updated the task description. (Show Details)May 1 2018, 4:34 PM

Completed auto-reimage of hosts:

['analytics1048.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1047.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011711_otto_6895.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1045.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011711_otto_6854.log.

Completed auto-reimage of hosts:

['analytics1046.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1045.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1044.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011851_otto_29689.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1043.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805011851_otto_29682.log.

Completed auto-reimage of hosts:

['analytics1044.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1043.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1042.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805021749_otto_8815.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1041.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805021749_otto_8808.log.

Completed auto-reimage of hosts:

['analytics1041.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1042.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1039.eqiad.wmnet', 'analytics1040.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805030559_elukey_31055.log.

Completed auto-reimage of hosts:

['analytics1039.eqiad.wmnet', 'analytics1040.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1037.eqiad.wmnet', 'analytics1038.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805070911_elukey_7323.log.

Completed auto-reimage of hosts:

['analytics1037.eqiad.wmnet', 'analytics1038.eqiad.wmnet']

and were ALL successful.

mforns set the point value for this task to 13.May 7 2018, 3:47 PM
mforns changed the point value for this task from 13 to 8.May 7 2018, 3:48 PM
Ottomata updated the task description. (Show Details)May 8 2018, 7:56 PM

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1036.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805082009_otto_31133.log.

Script wmf-auto-reimage was launched by otto on neodymium.eqiad.wmnet for hosts:

['analytics1034.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805082009_otto_31126.log.

Completed auto-reimage of hosts:

['analytics1034.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['analytics1036.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1032.eqiad.wmnet', 'analytics1033.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805091322_elukey_24654.log.

Completed auto-reimage of hosts:

['analytics1032.eqiad.wmnet', 'analytics1033.eqiad.wmnet']

and were ALL successful.

elukey moved this task from Backlog to In Progress on the User-Elukey board.May 10 2018, 11:43 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1030.eqiad.wmnet', 'analytics1031.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805101200_elukey_13942.log.

Completed auto-reimage of hosts:

['analytics1031.eqiad.wmnet', 'analytics1030.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1029.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805101343_elukey_18767.log.

Completed auto-reimage of hosts:

['analytics1029.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1028.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805101634_elukey_15122.log.

Completed auto-reimage of hosts:

['analytics1028.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1035.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805110812_elukey_17619.log.

Completed auto-reimage of hosts:

['analytics1035.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['analytics1052.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805111028_elukey_10112.log.

Completed auto-reimage of hosts:

['analytics1052.eqiad.wmnet']

and were ALL successful.

elukey@neodymium:~$ sudo cumin 'A:hadoop-worker' 'cat /etc/debian_version'
50 hosts will be targeted:
analytics[1028-1077].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(50) analytics[1028-1077].eqiad.wmnet
----- OUTPUT of 'cat /etc/debian_version' -----
9.4

Change 432564 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::worker: drop Debian Jessie support

https://gerrit.wikimedia.org/r/432564

elukey moved this task from In Progress to Done on the User-Elukey board.May 14 2018, 8:06 AM

Triple checking disk layout:

elukey@analytics1030:~$ df -h
Filesystem                          Size  Used Avail Use% Mounted on
udev                                 32G     0   32G   0% /dev
tmpfs                               6.3G   90M  6.3G   2% /run
/dev/mapper/analytics1030--vg-root   55G  6.2G   46G  12% /
tmpfs                                32G     0   32G   0% /dev/shm
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                                32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                           922M   37M  822M   5% /boot
/dev/sdf1                           3.6T  3.1T  524G  86% /var/lib/hadoop/data/f
/dev/sdd1                           3.6T  3.1T  525G  86% /var/lib/hadoop/data/d
/dev/sdm1                           3.6T  3.1T  521G  86% /var/lib/hadoop/data/m
/dev/sdb1                           3.6T  3.1T  526G  86% /var/lib/hadoop/data/b
/dev/sdk1                           3.6T  3.1T  525G  86% /var/lib/hadoop/data/k
/dev/sdi1                           3.6T  3.1T  520G  86% /var/lib/hadoop/data/i
/dev/sdg1                           3.6T  3.1T  525G  86% /var/lib/hadoop/data/g
/dev/sde1                           3.6T  3.1T  520G  86% /var/lib/hadoop/data/e
/dev/sdc1                           3.6T  3.1T  526G  86% /var/lib/hadoop/data/c
/dev/sdl1                           3.6T  3.1T  520G  86% /var/lib/hadoop/data/l
/dev/sdj1                           3.6T  3.1T  524G  86% /var/lib/hadoop/data/j
/dev/sdh1                           3.6T  3.1T  525G  86% /var/lib/hadoop/data/h
tmpfs                               6.3G     0  6.3G   0% /run/user/117
tmpfs                               6.3G     0  6.3G   0% /run/user/115
tmpfs                               6.3G     0  6.3G   0% /run/user/13926

elukey@analytics1030:~$ sudo lvs
  LV     VG               Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   analytics1030-vg -wi-ao----  55.88g
  swap   analytics1030-vg -wi-ao----   9.31g
  unused analytics1030-vg -wi-a----- 166.25g

elukey@analytics1030:~$ sudo pvs
  PV         VG               Fmt  Attr PSize   PFree
  /dev/sda5  analytics1030-vg lvm2 a--  231.44g    0

elukey@analytics1030:~$ sudo fdisk -l /dev/sda5
Disk /dev/sda5: 231.5 GiB, 248508317696 bytes, 485367808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

So /dev/sda5 is a disk in hw raid1 that contains a lvm physical volume and logical volumes on top of it. In df it is not indicated anymore as "dm-0" but as regular disk (less confusing than before for sure since there is not software raid involved).

Here's the raid 1 config:

elukey@analytics1030:~$ sudo megacli -LDInfo -Lall -aALL

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :Virtual Disk 0
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 232.375 GB
Sector Size         : 512
Mirror Data         : 232.375 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: Yes
LD has drives that support T10 power conditions: No
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only

@Ottomata docs should be updated with the new partitions layout but let me know if you see something that is odd.

elukey moved this task from In Progress to Done on the Analytics-Kanban board.May 14 2018, 8:12 AM

Docs look good elukey thank you!

Change 432564 merged by Elukey:
[operations/puppet@production] profile::hadoop::worker: drop Debian Jessie support

https://gerrit.wikimedia.org/r/432564

Nuria closed this task as Resolved.May 29 2018, 11:22 AM