Bring an-worker1[187-208] into the hadoop cluster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Stevemunene
	Mar 11 2025, 10:07 AM

Project Tags

Referenced Files

	F58859546: image.png
	Mar 18 2025, 8:35 AM

	F58817809: image.png
	Mar 13 2025, 3:46 PM

Subscribers

Aklapper

• Stevemunene

Description

We have new hosts that have been racked and setup on T386390, the next step is to bring them into the cluster which will increase our capacity and is a step towards getting ready for the hadoop 3 upgrade.

Assign the servers the right partman recipe analytics-flex.cfg (This was done during initial setup)
Install Debian Bullseye on the hosts.
Create the server's dummy keytabs.
Create the server's kerberos keytabs
Run the hadoop-init-worker.py cookbook to setup the remaining partitions
Setup journalnode on each
Add the hosts to net_topology with the right rack assignment
Add the hosts to role(analytics_cluster::hadoop::worker)

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
hdfs: Fix an-worker regex to include more hosts	Stevemunene	operations/puppet	production	+1 -1
hdfs: Assign the right role to new hdfs workers 1[187-208]	Stevemunene	operations/puppet	production	+2 -2
hdfs: Add new worker hosts1[187-208] to net_topology	Stevemunene	operations/puppet	production	+22 -0
hdfs: create dummy keytabs for new hosts	Stevemunene	labs/private	master	+0 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• Stevemunene	T388512 Bring an-worker1[187-208] into the hadoop cluster
		Resolved	Request	Jclark-ctr	T389065 hw troubleshooting: disk in slot 10 for an-worker1194

Event Timeline

• Stevemunene created this task.Mar 11 2025, 10:07 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2025, 10:07 AM

• Stevemunene updated the task description. (Show Details)Mar 11 2025, 10:07 AM

• Stevemunene moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.

• Stevemunene updated the task description. (Show Details)Mar 11 2025, 12:56 PM

• Stevemunene updated the task description. (Show Details)Mar 12 2025, 9:49 AM

Change #1126955 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[labs/private@master] hdfs: create dummy keytabs for new hosts

https://gerrit.wikimedia.org/r/1126955

gerritbot added a project: Patch-For-Review.Mar 12 2025, 11:09 AM

Change #1126957 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] hdfs: Add new worker hosts1[187-208] to net_topology

https://gerrit.wikimedia.org/r/1126957

Change #1126958 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] hdfs: Assign the right role to new hdfs workers 1[187-208]

https://gerrit.wikimedia.org/r/1126958

Change #1126955 merged by Stevemunene:

[labs/private@master] hdfs: create dummy keytabs for new hosts

https://gerrit.wikimedia.org/r/1126955

We have some errors on 2 hosts currently being looked into

an-worker1194.eqiad.wmnet fails the sre-init hadoop worker playbook which sets up the drives with an error on one disk bad superblock on /dev/sdl1
an-worker1204.eqiad.wnmet is unreachable

In T388512#10631626, @Stevemunene wrote:

We have some errors on 2 hosts currently being looked into

an-worker1194.eqiad.wmnet fails the sre-init hadoop worker playbook which sets up the drives with an error on one disk bad superblock on /dev/sdl1

an-worker1204.eqiad.wnmet is unreachable

an-worker1194 seems to have an error on disk sdl and sdb but sdb successfully mounts. The host is stuck booting

;1;39mFile System Check…/dev/disk/by-label/hadoop-l..

an-worker1204 is back online and ok

stevemunene@an-worker1204:~$ lsblk -i
NAME                               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                  8:0    0   7.3T  0 disk 
`-sda1                               8:1    0   7.3T  0 part /var/lib/hadoop/data/a
sdb                                  8:16   0   7.3T  0 disk 
`-sdb1                               8:17   0   7.3T  0 part /var/lib/hadoop/data/b
sdc                                  8:32   0   7.3T  0 disk 
`-sdc1                               8:33   0   7.3T  0 part /var/lib/hadoop/data/c
sdd                                  8:48   0   7.3T  0 disk 
`-sdd1                               8:49   0   7.3T  0 part /var/lib/hadoop/data/d
sde                                  8:64   0   7.3T  0 disk 
`-sde1                               8:65   0   7.3T  0 part /var/lib/hadoop/data/e
sdf                                  8:80   0   7.3T  0 disk 
`-sdf1                               8:81   0   7.3T  0 part /var/lib/hadoop/data/f
sdg                                  8:96   0   7.3T  0 disk 
`-sdg1                               8:97   0   7.3T  0 part /var/lib/hadoop/data/g
sdh                                  8:112  0   7.3T  0 disk 
`-sdh1                               8:113  0   7.3T  0 part /var/lib/hadoop/data/h
sdi                                  8:128  0   7.3T  0 disk 
`-sdi1                               8:129  0   7.3T  0 part /var/lib/hadoop/data/i
sdj                                  8:144  0   7.3T  0 disk 
`-sdj1                               8:145  0   7.3T  0 part /var/lib/hadoop/data/j
sdk                                  8:160  0   7.3T  0 disk 
`-sdk1                               8:161  0   7.3T  0 part /var/lib/hadoop/data/k
sdl                                  8:176  0   7.3T  0 disk 
`-sdl1                               8:177  0   7.3T  0 part /var/lib/hadoop/data/l
sdm                                  8:192  0 446.6G  0 disk 
|-sdm1                               8:193  0   953M  0 part /boot
|-sdm2                               8:194  0     1K  0 part 
`-sdm5                               8:197  0 445.7G  0 part 
  |-an--worker1204--vg-swap        254:0    0   9.3G  0 lvm  [SWAP]
  |-an--worker1204--vg-root        254:1    0  55.9G  0 lvm  /
  `-an--worker1204--vg-journalnode 254:2    0    10G  0 lvm  /var/lib/hadoop/journal

Change #1126957 merged by Stevemunene:

[operations/puppet@production] hdfs: Add new worker hosts1[187-208] to net_topology

https://gerrit.wikimedia.org/r/1126957

• Stevemunene updated the task description. (Show Details)Mar 13 2025, 9:54 AM

Mentioned in SAL (#wikimedia-analytics) [2025-03-13T10:50:50Z] <stevemunene> disable puppet on an-worker1[187-208] to bring them into the hadoop cluster in batches T388512

Change #1126958 merged by Stevemunene:

[operations/puppet@production] hdfs: Assign the right role to new hdfs workers 1[187-208]

https://gerrit.wikimedia.org/r/1126958

Mentioned in SAL (#wikimedia-analytics) [2025-03-13T11:03:41Z] <stevemunene> roll restart hadoop master hosts to pick up new hosts an-worker1[187-208] T388512

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2025, 11:30 AM

Mentioned in SAL (#wikimedia-analytics) [2025-03-13T13:48:46Z] <stevemunene> restart the hadoop-hdfs-namenode service on an-master1004 to pick up the new hosts as well T388512

Change #1127532 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] hdfs: Fix an-worker regex to include more hosts

https://gerrit.wikimedia.org/r/1127532

gerritbot added a project: Patch-For-Review.Mar 13 2025, 2:24 PM

Change #1127532 merged by Stevemunene:

[operations/puppet@production] hdfs: Fix an-worker regex to include more hosts

https://gerrit.wikimedia.org/r/1127532

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2025, 3:30 PM

New hosts onboarding in batches

• Stevemunene moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 13 2025, 3:46 PM

• Stevemunene triaged this task as High priority.Mar 14 2025, 6:26 AM

• Stevemunene updated the task description. (Show Details)

Icinga downtime and Alertmanager silence (ID=209b3662-ef42-401c-9bca-58fe9982d708) set by stevemunene@cumin1002 for 3 days, 0:00:00 on 9 host(s) and their services with reason: Adding the hosts to the analytics hadoop cluster in batches. this is part of the next batch

an-worker[1200-1208].eqiad.wmnet

Icinga downtime and Alertmanager silence (ID=6096d014-7472-4b75-9a92-cb6f42eecd34) set by stevemunene@cumin1002 for 3 days, 0:00:00 on 1 host(s) and their services with reason: Adding the hosts to the analytics hadoop cluster in batches. this is part of the next batch