Page MenuHomePhabricator
Create Task
Maniphest T237269

Prepare the Hadoop Analytics cluster for Kerberos
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

This task should list all the steps to take before enabling Kerberos:

  • Create user principals for various teams - T237605
  • Create and deploy system users and their keytabs to hosts (like analytics-search, etc..)
  • T238306
  • Kerberize all the nodes that will need it, and deploy keytabs generated in the above step (This can be done separately and it is good to test if kerberos works on all the nodes).
  • Add analytics users (without SSH access) to all Hadoop worker nodes
  • Add TLS keys for Master nodes if needed (may be needed only for workers, to be checked)
  • Add druid and analytics search system users to all Hadoop worker nodes
  • Disable Meta DB backup to HDFS
  • T231208
  • Find a solution for labstore crons
  • T234229
  • Create Oozie automation to get a snapshot of the current jobs and restart them
  • T237271

Details

SubjectRepoBranchLines +/-
Put hive-site.xml into HDFS as analytics useroperations/puppetproduction+3 -2
Update hive and spark oozie jobs for kerberosanalytics/refinerymaster+1 K -90
Set g+s on all yarn user cache directoriesoperations/puppetproduction+11 -3
Enable Kerberos in Hadoop Analytics and Druid Analytics/Publicoperations/puppetproduction+136 -2
profile::hadoop::worker: add set_yarn_dir_ownership scriptoperations/puppetproduction+63 -0
Add hdfs kerberos keytab to Analytics Hadoop coordinatorsoperations/puppetproduction+11 -1
role::analytics_test_cluster::coordinator: fix Presto kerberos configoperations/puppetproduction+8 -4
role::analytics_test_cluster_coordinator: enable Kerberos for Prestooperations/puppetproduction+9 -0
profile::analytics::cluster::client: fix nagios' sudo permissionsoperations/puppetproduction+1 -1
hadoop: add analytics kerberos keytab for test/prod master hostoperations/puppetproduction+11 -1
Deploy Kerberos keytabs for analytics-search and presto system usersoperations/puppetproduction+9 -1
profile::kerberos::client: add MOTD to help usersoperations/puppetproduction+22 -0
Deploy kerberos keytabs on stat100[4,5,7]operations/puppetproduction+18 -0
Deploy Kerberos keytabs to Analytics Hadoop hostsoperations/puppetproduction+97 -2
Include Kerberos profiles in the Analytics infrastructureoperations/puppetproduction+25 -0
Add TLS certificates to Hadoop Analytics master nodesoperations/puppetproduction+6 -0
Add druid/analytics/search system users to all Hadoop worker nodesoperations/puppetproduction+1 -0
Add analytics users (without ssh keys) to all Hadoop worker nodesoperations/puppetproduction+7 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
elukey updated the task description. (Show Details)Nov 4 2019, 3:21 PM
elukey updated the task description. (Show Details)Nov 4 2019, 3:26 PM
gerritbot added a comment.Nov 4 2019, 3:35 PM

Change 548293 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add analytics users (without ssh keys) to all Hadoop worker nodes

https://gerrit.wikimedia.org/r/548293

gerritbot added a project: Patch-For-Review.Nov 4 2019, 3:35 PM
gerritbot added a comment.Nov 4 2019, 3:40 PM

Change 548294 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add druid/analytics/search system users to all Hadoop worker nodes

https://gerrit.wikimedia.org/r/548294

• fdans triaged this task as High priority.Nov 4 2019, 4:43 PM
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.
gerritbot added a comment.Nov 5 2019, 9:14 AM

Change 548293 merged by Elukey:
[operations/puppet@production] Add analytics users (without ssh keys) to all Hadoop worker nodes

https://gerrit.wikimedia.org/r/548293

gerritbot added a comment.Nov 5 2019, 9:17 AM

Change 548294 merged by Elukey:
[operations/puppet@production] Add druid/analytics/search system users to all Hadoop worker nodes

https://gerrit.wikimedia.org/r/548294

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2019, 10:10 AM
elukey updated the task description. (Show Details)Nov 5 2019, 10:27 AM
• MoritzMuehlenhoff subscribed.Nov 5 2019, 10:40 AM
elukey updated the task description. (Show Details)Nov 5 2019, 2:04 PM
elukey updated the task description. (Show Details)Nov 5 2019, 2:41 PM
elukey updated the task description. (Show Details)
gerritbot added a comment.Nov 5 2019, 3:04 PM

Change 548759 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add TLS certificates to Hadoop Analytics master nodes

https://gerrit.wikimedia.org/r/548759

gerritbot added a project: Patch-For-Review.Nov 5 2019, 3:04 PM
gerritbot added a comment.Nov 5 2019, 3:07 PM

Change 548759 merged by Elukey:
[operations/puppet@production] Add TLS certificates to Hadoop Analytics master nodes

https://gerrit.wikimedia.org/r/548759

elukey updated the task description. (Show Details)Nov 5 2019, 3:09 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 5 2019, 3:10 PM
elukey updated the task description. (Show Details)Nov 6 2019, 8:59 AM
gerritbot added a comment.Nov 6 2019, 1:31 PM

Change 549083 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Include Kerberos profiles in the Analytics infrastructure

https://gerrit.wikimedia.org/r/549083

gerritbot added a project: Patch-For-Review.Nov 6 2019, 1:31 PM
gerritbot added a comment.Nov 6 2019, 2:13 PM

Change 549083 merged by Elukey:
[operations/puppet@production] Include Kerberos profiles in the Analytics infrastructure

https://gerrit.wikimedia.org/r/549083

elukey updated the task description. (Show Details)Nov 6 2019, 2:17 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 6 2019, 3:10 PM
gerritbot added a comment.Nov 6 2019, 3:51 PM

Change 549103 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Deploy Kerberos keytabs to Analytics Hadoop hosts

https://gerrit.wikimedia.org/r/549103

gerritbot added a project: Patch-For-Review.Nov 6 2019, 3:51 PM
gerritbot added a comment.Nov 6 2019, 4:32 PM

Change 549103 merged by Elukey:
[operations/puppet@production] Deploy Kerberos keytabs to Analytics Hadoop hosts

https://gerrit.wikimedia.org/r/549103

elukey updated the task description. (Show Details)Nov 6 2019, 4:43 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 6 2019, 5:10 PM
elukey updated the task description. (Show Details)Nov 7 2019, 6:50 AM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Nov 7 2019, 6:57 AM
gerritbot added a comment.Nov 7 2019, 4:23 PM

Change 549565 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Deploy kerberos keytabs on stat100[4,5,7]

https://gerrit.wikimedia.org/r/549565

gerritbot added a project: Patch-For-Review.Nov 7 2019, 4:23 PM

Change 549565 merged by Elukey:
[operations/puppet@production] Deploy kerberos keytabs on stat100[4,5,7]

https://gerrit.wikimedia.org/r/549565

gerritbot added a comment.Nov 7 2019, 4:34 PM

Change 549566 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable Kerberos in Hadoop Analytics and Druid Analytics/Public

https://gerrit.wikimedia.org/r/549566

elukey updated the task description. (Show Details)Nov 7 2019, 4:46 PM
elukey updated the task description. (Show Details)Nov 7 2019, 4:59 PM
elukey updated the task description. (Show Details)Nov 8 2019, 7:17 AM
gerritbot added a comment.Nov 11 2019, 9:32 AM

Change 550099 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kerberos::client: add MOTD to help users

https://gerrit.wikimedia.org/r/550099

gerritbot added a comment.Nov 11 2019, 9:39 AM

Change 550099 merged by Elukey:
[operations/puppet@production] profile::kerberos::client: add MOTD to help users

https://gerrit.wikimedia.org/r/550099

elukey updated the task description. (Show Details)Nov 13 2019, 10:23 AM
gerritbot added a comment.Nov 13 2019, 4:37 PM

Change 550713 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Deploy Kerberos keytabs for analytics-search and presto system users

https://gerrit.wikimedia.org/r/550713

gerritbot added a comment.Nov 13 2019, 4:39 PM

Change 550713 merged by Elukey:
[operations/puppet@production] Deploy Kerberos keytabs for analytics-search and presto system users

https://gerrit.wikimedia.org/r/550713

elukey updated the task description. (Show Details)Nov 14 2019, 5:46 PM
gerritbot added a comment.Nov 14 2019, 9:43 PM

Change 550945 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update hive and spark oozie jobs for kerberos

https://gerrit.wikimedia.org/r/550945

elukey updated the task description. (Show Details)Nov 15 2019, 9:33 AM
gerritbot added a comment.Nov 18 2019, 7:54 AM

Change 551392 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hadoop: add analytics kerberos keytab for test/prod master host

https://gerrit.wikimedia.org/r/551392

gerritbot added a comment.Nov 18 2019, 7:56 AM

Change 551392 merged by Elukey:
[operations/puppet@production] hadoop: add analytics kerberos keytab for test/prod master host

https://gerrit.wikimedia.org/r/551392

gerritbot added a comment.Nov 18 2019, 8:18 AM

Change 551398 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::cluster::client: fix nagios' sudo permissions

https://gerrit.wikimedia.org/r/551398

gerritbot added a comment.Nov 18 2019, 8:22 AM

Change 551398 merged by Elukey:
[operations/puppet@production] profile::analytics::cluster::client: fix nagios' sudo permissions

https://gerrit.wikimedia.org/r/551398

elukey updated the task description. (Show Details)Nov 18 2019, 9:31 AM
elukey added a project: User-Elukey.Nov 19 2019, 2:59 PM
elukey added a comment.Nov 19 2019, 4:16 PM

I was able to make Presto run with the following config settings on analytics1030:

/etc/presto/catalog/analytics_test_hive.properties

hive.metastore.authentication.type=KERBEROS
hive.metastore.service.principal=hive/analytics1030.eqiad.wmnet@WIKIMEDIA
hive.metastore.client.principal=presto/analytics1030.eqiad.wmnet@WIKIMEDIA
hive.metastore.client.keytab=/etc/security/keytabs/presto/presto.keytab

hive.hdfs.authentication.type=KERBEROS
hive.hdfs.impersonation.enabled=true
hive.hdfs.presto.principal=presto/analytics1030.eqiad.wmnet@WIKIMEDIA
hive.hdfs.presto.keytab=/etc/security/keytabs/presto/presto.keytab
hive.hdfs.impersonation.enabled=true
hive.hdfs.wire-encryption.enabled=true
/etc/presto/config.properties

http-server.authentication.type=KERBEROS
http.server.authentication.krb5.service-name=presto
http.server.authentication.krb5.keytab=/etc/security/keytabs/presto/presto.keytab
http.authentication.krb5.config=/etc/krb5.conf

TLS seems optional, and it makes sense, since the kerberos auth is protected, but then the traffic between user and presto coord is unencrypted. I think that we can live with this for the moment, and possibly review it after Kerberos is enabled?

gerritbot added a comment.Nov 19 2019, 5:29 PM

Change 551870 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster_coordinator: enable Kerberos for Presto

https://gerrit.wikimedia.org/r/551870

gerritbot added a comment.Nov 19 2019, 5:35 PM

Change 551870 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster_coordinator: enable Kerberos for Presto

https://gerrit.wikimedia.org/r/551870

gerritbot added a comment.Nov 19 2019, 5:53 PM

Change 551881 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordinator: fix Presto kerberos config

https://gerrit.wikimedia.org/r/551881

gerritbot added a comment.Nov 19 2019, 5:55 PM

Change 551881 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordinator: fix Presto kerberos config

https://gerrit.wikimedia.org/r/551881

elukey updated the task description. (Show Details)Nov 19 2019, 6:11 PM
gerritbot added a comment.Nov 20 2019, 7:19 AM

Change 551957 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add hdfs kerberos keytab to Analytics Hadoop coordinators

https://gerrit.wikimedia.org/r/551957

gerritbot added a comment.Nov 20 2019, 7:23 AM

Change 551957 merged by Elukey:
[operations/puppet@production] Add hdfs kerberos keytab to Analytics Hadoop coordinators

https://gerrit.wikimedia.org/r/551957

jbond closed subtask T238306: Add system user analytics-privatedata to the anaytics-privatedata-users group as Resolved.Nov 27 2019, 11:57 AM
gerritbot added a comment.Dec 10 2019, 1:55 PM

Change 556190 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::worker: add set_yarn_dir_ownership script

https://gerrit.wikimedia.org/r/556190

gerritbot added a comment.Dec 10 2019, 2:55 PM

Change 556190 merged by Elukey:
[operations/puppet@production] profile::hadoop::worker: add set_yarn_dir_ownership script

https://gerrit.wikimedia.org/r/556190

gerritbot added a comment.Dec 16 2019, 1:16 PM

Change 549566 merged by Elukey:
[operations/puppet@production] Enable Kerberos in Hadoop Analytics and Druid Analytics/Public

https://gerrit.wikimedia.org/r/549566

gerritbot added a comment.Dec 16 2019, 2:54 PM

Change 558067 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set g+s on all yarn user cache directories

https://gerrit.wikimedia.org/r/558067

gerritbot added a comment.Dec 16 2019, 2:57 PM

Change 558067 merged by Ottomata:
[operations/puppet@production] Set g+s on all yarn user cache directories

https://gerrit.wikimedia.org/r/558067

gerritbot added a comment.Dec 16 2019, 3:14 PM

Change 550945 merged by Mforns:
[analytics/refinery@master] Update hive and spark oozie jobs for kerberos

https://gerrit.wikimedia.org/r/550945

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2019, 4:10 PM
gerritbot added a comment.Dec 16 2019, 5:08 PM

Change 558128 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Put hive-site.xml into HDFS as analytics user

https://gerrit.wikimedia.org/r/558128

gerritbot added a project: Patch-For-Review.Dec 16 2019, 5:08 PM
gerritbot added a comment.Dec 16 2019, 5:13 PM

Change 558128 merged by Ottomata:
[operations/puppet@production] Put hive-site.xml into HDFS as analytics user

https://gerrit.wikimedia.org/r/558128

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2019, 6:11 PM
elukey set the point value for this task to 21.Dec 17 2019, 9:32 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
elukey closed subtask T238560: Doubts and questions about Kerberos and Hadoop as Resolved.Dec 18 2019, 11:07 AM
• Nuria closed subtask T237271: Create a script to ease the Oozie work while enabling kerberos in Hadoop as Resolved.Dec 20 2019, 5:28 PM
• chasemp closed subtask T237605: Create kerberos principals for users as Resolved.Dec 23 2019, 4:40 PM
elukey moved this task from Backlog to Kerberos on the User-Elukey board.Jan 3 2020, 10:56 AM
elukey moved this task from Kerberos to Done on the User-Elukey board.
Ottomata reopened subtask T238560: Doubts and questions about Kerberos and Hadoop as Open.Jan 6 2020, 8:26 PM
• Nuria closed this task as Resolved.Jan 7 2020, 2:00 PM
elukey closed subtask T238560: Doubts and questions about Kerberos and Hadoop as Resolved.Jan 7 2020, 5:12 PM
• Nuria closed subtask T239903: Kerberize Superset to allow Presto queries as Resolved.Apr 9 2020, 3:12 PM
• Nuria closed subtask T240880: profile::hive::site_hdfs should work with kerberos-run-command as Resolved.Apr 9 2020, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits