Page MenuHomePhabricator
Create Task
Maniphest T236995

Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

This task is about enabling TLS encryption and authentication for the MapReduce shufflers in the Hadoop Analytics cluster. We have already running it in the test cluster for months without any issues registered so far.

In a nutshell, during the shuffle step of a mapreduce job the reducer tasks pull data from the mappers via a Yarn service called MapReduce shuffler, using HTTP. PII data might be flowing so we want to secure that traffic, see:

https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html

This is a pre-step before enabling RPC encryption and Kerberos!

Things to do:

  • Allow cergen to specify a java trustore password in its config, rather than use the key's password.
  • Generate TLS configs and keys in the Puppet private repo via cergen
  • Deploy TLS mapreduce-ssl xml configs and TLS certs to the Hadoop worker nodes - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/547491/
  • Enable TLS for the mapreduce shufflers (requires node manager roll restart)

Details

SubjectRepoBranchLines +/-
Enable mapreduce.ssl.enabled in Hadoop Analyticsoperations/puppetproduction+1 -0
Enable the encrypted shuffle functionality in Hadoop Analyticsoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Oct 31 2019, 10:47 AM
gerritbot added a comment.Oct 31 2019, 12:29 PM

Change 547522 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable the encrypted shuffle functionality in Hadoop Analytics

https://gerrit.wikimedia.org/r/547522

gerritbot added a project: Patch-For-Review.Oct 31 2019, 12:29 PM
elukey added a project: Analytics-Kanban.Oct 31 2019, 2:16 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey added a comment.Oct 31 2019, 2:27 PM

The procedure should be as simple as:

  1. temporarily stop timers on an-coordinator and let the cluster drain
  2. merge the above code patch, run puppet
  3. roll restart all Yarn node managers

To verify that a node is working after the change: echo y | openssl s_client -connect localhost:13562

gerritbot added a comment.Oct 31 2019, 2:39 PM

Change 547522 merged by Elukey:
[operations/puppet@production] Enable the encrypted shuffle functionality in Hadoop Analytics

https://gerrit.wikimedia.org/r/547522

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2019, 3:10 PM
gerritbot added a comment.Oct 31 2019, 3:24 PM

Change 547557 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable mapreduce.ssl.enabled in Hadoop Analytics

https://gerrit.wikimedia.org/r/547557

gerritbot added a project: Patch-For-Review.Oct 31 2019, 3:24 PM

Change 547557 merged by Elukey:
[operations/puppet@production] Enable mapreduce.ssl.enabled in Hadoop Analytics

https://gerrit.wikimedia.org/r/547557

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2019, 4:10 PM
Ottomata triaged this task as High priority.Oct 31 2019, 4:36 PM
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.
elukey added a comment.Oct 31 2019, 5:02 PM

Summary of things done if a rollback is needed:

mapred_site_extra_properties:
  mapreduce.ssl.enabled: true
  mapreduce.shuffle.ssl.enabled: true

core_site_extra_properties:
  hadoop.ssl.enabled.protocols: 'TLSv1.2'

mapreduce.ssl.enabled: true needed to be on an-coord1001 and I had to restart hive-server2 and oozie to pick up the new option.

elukey added a subscriber: Ottomata.Nov 4 2019, 8:38 AM

Added documentation to https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#New_Worker_Installation_(12_disk,_2_flex_bay_drives_-_analytics1028-analytics1077) Cc: @Ottomata

elukey updated the task description. (Show Details)Nov 4 2019, 8:39 AM
elukey set the point value for this task to 8.
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
• Nuria closed this task as Resolved.Nov 7 2019, 11:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL