Page MenuHomePhabricator

Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster
Closed, ResolvedPublic8 Estimated Story Points

Description

This task is about enabling TLS encryption and authentication for the MapReduce shufflers in the Hadoop Analytics cluster. We have already running it in the test cluster for months without any issues registered so far.

In a nutshell, during the shuffle step of a mapreduce job the reducer tasks pull data from the mappers via a Yarn service called MapReduce shuffler, using HTTP. PII data might be flowing so we want to secure that traffic, see:

https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html

This is a pre-step before enabling RPC encryption and Kerberos!

Things to do:

  • Allow cergen to specify a java trustore password in its config, rather than use the key's password.
  • Generate TLS configs and keys in the Puppet private repo via cergen
  • Deploy TLS mapreduce-ssl xml configs and TLS certs to the Hadoop worker nodes - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/547491/
  • Enable TLS for the mapreduce shufflers (requires node manager roll restart)

Event Timeline

elukey created this task.Oct 31 2019, 10:47 AM

Change 547522 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable the encrypted shuffle functionality in Hadoop Analytics

https://gerrit.wikimedia.org/r/547522

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

The procedure should be as simple as:

  1. temporarily stop timers on an-coordinator and let the cluster drain
  2. merge the above code patch, run puppet
  3. roll restart all Yarn node managers

To verify that a node is working after the change: echo y | openssl s_client -connect localhost:13562

Change 547522 merged by Elukey:
[operations/puppet@production] Enable the encrypted shuffle functionality in Hadoop Analytics

https://gerrit.wikimedia.org/r/547522

Change 547557 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable mapreduce.ssl.enabled in Hadoop Analytics

https://gerrit.wikimedia.org/r/547557

Change 547557 merged by Elukey:
[operations/puppet@production] Enable mapreduce.ssl.enabled in Hadoop Analytics

https://gerrit.wikimedia.org/r/547557

Ottomata triaged this task as High priority.Oct 31 2019, 4:36 PM
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.

Summary of things done if a rollback is needed:

mapred_site_extra_properties:
  mapreduce.ssl.enabled: true
  mapreduce.shuffle.ssl.enabled: true

core_site_extra_properties:
  hadoop.ssl.enabled.protocols: 'TLSv1.2'

mapreduce.ssl.enabled: true needed to be on an-coord1001 and I had to restart hive-server2 and oozie to pick up the new option.

elukey updated the task description. (Show Details)Nov 4 2019, 8:39 AM
elukey set the point value for this task to 8.
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
Nuria closed this task as Resolved.Nov 7 2019, 11:13 PM