Page MenuHomePhabricator
Create Task
Maniphest T343762

Bring Hadoop workers an-worker11[49-56] into service
Closed, ResolvedPublic
Actions

Description

We have 8 new Hadoop workers to bring into service. These were racked in T327295: Q3:rack/setup/install an-worker11[49-56] and purchased in T325206.
For these we are going to:

  • Assign the servers the right partman recipe analytics-flex.cfg
  • Install Debian Bullseye on the hosts.
  • Run the hadoop-init-worker.py cookbook to setup the remaining partitions
  • Create the server's kerberos keytabs.
  • Add the hosts to role(analytics_cluster::hadoop::worker)

Details

SubjectRepoBranchLines +/-
Register hadoop workers an-worker-1149->1156.eqiad.wmnetoperations/puppetproduction+12 -8
Add dummy secrets for all new an-worker11(49->56).eqiad.wmnet hadoop workerslabs/privatemaster+0 -0
Correct the role for the new hadoop workersoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

BTullis created this task.Aug 7 2023, 9:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2023, 9:51 PM
BTullis mentioned this in T343763: Decommission analytics10[70-77].Aug 7 2023, 9:53 PM
gerritbot added a comment.Aug 8 2023, 5:05 PM

Change 946978 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Correct the role for the new hadoop workers

https://gerrit.wikimedia.org/r/946978

gerritbot added a project: Patch-For-Review.Aug 8 2023, 5:05 PM
gerritbot added a comment.Aug 8 2023, 5:19 PM

Change 946978 merged by Btullis:

[operations/puppet@production] Correct the role for the new hadoop workers

https://gerrit.wikimedia.org/r/946978

Maintenance_bot removed a project: Patch-For-Review.Aug 8 2023, 5:30 PM
Gehel moved this task from Incoming to Misc on the Data-Platform-SRE board.Aug 18 2023, 8:33 AM
BTullis moved this task from Misc to Ready for Work on the Data-Platform-SRE board.Aug 22 2023, 3:39 PM
Gehel moved this task from Ready for Work to In Progress on the Data-Platform-SRE board.Sep 1 2023, 8:37 AM
Stevemunene updated the task description. (Show Details)Sep 1 2023, 12:49 PM
brouberol claimed this task.Sep 11 2023, 3:02 PM
brouberol added a comment.Sep 11 2023, 3:48 PM

The disks were not mounted on these 8 hosts. I ran a megacli command on all hosts to fix this the situation:

brouberol@cumin1001:~$ sudo cumin 'an-worker11[49-56].eqiad.wmnet' 'sudo megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0'
brouberol added a comment.EditedSep 11 2023, 3:52 PM

an-worker1149.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1150.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1151.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1152.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1153.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1154eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1155.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

an-worker1156.eqiad.wmnet

  • Setup journal node
  • Create kerberos keytabs
  • Commit kerberos keytabs in puppet
  • Run sre.hadoop-init-workers cookbook

When everything is done:

  • Add the hosts to role(analytics_cluster::hadoop::worker)
brouberol updated the task description. (Show Details)Sep 12 2023, 7:59 AM
gerritbot added a comment.Sep 12 2023, 8:15 AM

Change 956785 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Register hadoop workers an-worker-1149->1156.eqiad.wmnet

https://gerrit.wikimedia.org/r/956785

gerritbot added a project: Patch-For-Review.Sep 12 2023, 8:15 AM
brouberol updated the task description. (Show Details)Sep 12 2023, 8:15 AM
gerritbot added a comment.Sep 12 2023, 8:40 AM

Change 956789 had a related patch set uploaded (by Brouberol; author: Brouberol):

[labs/private@master] Add dummy secrets for all new an-worker11(49->56).eqiad.wmnet hadoop workers

https://gerrit.wikimedia.org/r/956789

brouberol added a comment.Sep 12 2023, 8:40 AM
~/wmf/private master ❯ mkdir -p modules/secret/secrets/kerberos/keytabs/an-worker11{49,50,51,52,53,54,55,56}.eqiad.wmnet/hadoop
~/wmf/private master ❯ touch modules/secret/secrets/kerberos/keytabs/an-worker11{49,50,51,52,53,54,55,56}.eqiad.wmnet/hadoop/{hdfs,HTTP,yarn}.keytab
gerritbot added a comment.Sep 12 2023, 9:48 AM

Change 956789 merged by Brouberol:

[labs/private@master] Add dummy secrets for all new an-worker11(49->56).eqiad.wmnet hadoop workers

https://gerrit.wikimedia.org/r/956789

brouberol mentioned this in rLPRI4405d64b6e4c: Add dummy secrets for all new an-worker11(49->56).eqiad.wmnet hadoop workers.Sep 12 2023, 9:58 AM
gerritbot added a comment.Sep 12 2023, 11:12 AM

Change 956785 merged by Brouberol:

[operations/puppet@production] Register hadoop workers an-worker-1149->1156.eqiad.wmnet

https://gerrit.wikimedia.org/r/956785

brouberol added a comment.Sep 12 2023, 11:15 AM
brouberol@puppetmaster1001:~$ sudo puppet-merge
...
Brouberol: Register hadoop workers an-worker-1149->1156.eqiad.wmnet (1d71a9e090)
Merge these changes? (yes/no)? yes
Stashbot added a comment.Sep 12 2023, 11:21 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-12T11:21:25Z] <btullis> demonstrated the use of SAL for T343762

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 11:30 AM
brouberol added a comment.Sep 12 2023, 11:50 AM

It took 2 puppet runs to get hadoop-hdfs-datanode.service running on an-worker1149. I've set an icinga downtime on the remaining 7 hosts to avoid getting systemd service failure errors reported to us while puppet runs.

brouberol@cumin1001:~$ sudo cookbook sre.hosts.downtime -r 'Mute initial failures of hadoop-hdfs-datanode.service' 'an-worker115[0-6].eqiad.wmnet'
brouberol added a comment.Sep 12 2023, 12:10 PM

image.png (613×1 px, 121 KB)

All 8 workers have registered into the cluster. We now need to rolling-restart the master nodes to assign them to a non-default rack.

brouberol added a comment.Sep 12 2023, 12:41 PM
brouberol@cumin1001:~$ sudo cookbook sre.hadoop.roll-restart-masters analytics
brouberol closed this task as Resolved.Sep 12 2023, 2:59 PM

All workers are provisioned, registered in Hadoop and have their proper rack assignation. The system should rebalance HDFS blocks automatically.

brouberol moved this task from In Progress to Done on the Data-Platform-SRE board.Nov 6 2023, 1:36 PM
BTullis mentioned this in T353776: Bring an-worker11[57-75] into service.Jan 30 2024, 12:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL