Page MenuHomePhabricator
Create Task
Maniphest T211836

Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This is the parent task to track all the work needed to:

  • enable a stronger authentication mechanism to Analytics Hadoop (based on Kerberos) for all the users (automated and human) interacting with it.
  • enable authentication and data encryption for all the Hadoop protocols that carry sensitive/important data.

The two macro areas should be doable in isolation, to avoid enabling too many things at the same time.

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedelukeyT211836 Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services
 ResolvedelukeyT212256 Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one.
 ResolvedelukeyT212257 Set up a Kerberos KDC service in production with minimal puppet automation
ResolvedelukeyT212259 Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings.
Restricted Task
Restricted Task
 ResolvedelukeyT226232 Update the Camus checker to be able to authenticate via Kerberos
 ResolvedelukeyT216238 eqiad: (1) Ganeti VM for testing Kerberos in Production
 ResolvedelukeyT212258 Create test Kerberos identities/accounts for some selected users in hadoop test cluster
 ResolvedelukeyT217412 Enable encryption and authentication for TLS-based Hadoop services
 ResolvedelukeyT226089 Make the Kerberos infrastructure production ready
 ResolvedRobHT227288 eqiad: 1 misc node for the Kerberos KDC service
 ResolvedelukeyT233141 setup/install krb1001/WMF5173
 Resolved• CmjohnsonT233642 apply hostname labels for krb1001/WMF5173
 ResolvedNoneT227425 codfw: 1 misc node for the Kerberos KDC service
 ResolvedPapaulT233142 setup/install krb2001/WMF6577
 ResolvedPapaulT233962 apply hostname labels for krb2001/WMF6577
 ResolvedelukeyT234600 Decommission kerberos1001
 ResolvedelukeyT226104 Set up a generic workflow to create Kerberos accounts
 ResolvedelukeyT226698 Allow all Analytics tools to work with Kerberos auth
 ResolvedelukeyT226844 Create an-tool1006, a ganeti vm to be used as client for the Hadoop test cluster
 ResolvedelukeyT227257 Move refinery to hive 2 actions
 Resolved• NuriaT136858 Beeline does not print full stack traces when a query fails
 ResolvedOttomataT228291 Refine should accept principal name for hive2 jdbc connection for DDL
 ResolvedelukeyT234229 Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts
Resolved• CmjohnsonT227485 Decommission analytics10[28-31,33-41]
 Resolved• CmjohnsonT233080 Decommission analytics1032
 ResolvedelukeyT235418 Add metadata to puppet about kerberos accounts
 ResolvedelukeyT236995 Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster
 ResolvedelukeyT237269 Prepare the Hadoop Analytics cluster for Kerberos
 ResolvedJAllemandouT237271 Create a script to ease the Oozie work while enabling kerberos in Hadoop
 ResolvedelukeyT237605 Create kerberos principals for users
 ResolvedelukeyT238306 Add system user analytics-privatedata to the anaytics-privatedata-users group
 ResolvedelukeyT238560 Doubts and questions about Kerberos and Hadoop
 ResolvedelukeyT239903 Kerberize Superset to allow Presto queries
 ResolvedcchenT247329 Learn how to make dashboard on top of data on hadoop/hive via presto
 ResolvedelukeyT240880 profile::hive::site_hdfs should work with kerberos-run-command

Event Timeline

• Nuria created this task.Dec 13 2018, 12:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2018, 12:23 AM
• Nuria renamed this task from Kerberos Stand Alone service running in production to Kerberos stand alone service running in production.Dec 13 2018, 12:54 AM
• fdans assigned this task to elukey.Dec 13 2018, 5:51 PM
• fdans triaged this task as High priority.
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.
• Nuria added a subtask: T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..Dec 18 2018, 9:32 PM
• Nuria renamed this task from Kerberos stand alone service running in production to Kerberos service running in production.Dec 18 2018, 9:37 PM
• Nuria updated the task description. (Show Details)
• Nuria set the point value for this task to 0.
Krenair subscribed.Dec 18 2018, 9:40 PM
• Nuria added a parent task: Restricted Task.Dec 18 2018, 9:41 PM
• Nuria mentioned this in T198227: Prototype in labs new security measures for cluster.Dec 18 2018, 9:43 PM
elukey added a project: User-Elukey.Jan 3 2019, 9:35 AM
elukey moved this task from Backlog to In Progress on the User-Elukey board.Jan 7 2019, 8:23 AM
• Nuria closed subtask T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one. as Resolved.Feb 14 2019, 5:09 AM
elukey renamed this task from Kerberos service running in production to Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services.Mar 1 2019, 2:59 PM
elukey updated the task description. (Show Details)
elukey moved this task from In Progress to Keep an eye on it on the User-Elukey board.Mar 4 2019, 9:34 AM
mforns lowered the priority of this task from High to Medium.Mar 11 2019, 3:31 PM
elukey moved this task from Keep an eye on it to In Progress on the User-Elukey board.Jun 6 2019, 4:21 PM
• Nuria closed subtask T212257: Set up a Kerberos KDC service in production with minimal puppet automation as Resolved.Jun 24 2019, 9:10 PM
elukey moved this task from In Progress to Kerberos on the User-Elukey board.Jul 5 2019, 6:56 AM
elukey removed a project: User-Elukey.Jul 8 2019, 7:04 AM
elukey added a subtask: T227485: Decommission analytics10[28-31,33-41].Jul 8 2019, 2:34 PM
elukey changed the status of subtask T227485: Decommission analytics10[28-31,33-41] from Open to Stalled.
• Nuria closed subtask T217412: Enable encryption and authentication for TLS-based Hadoop services as Resolved.Jul 18 2019, 8:37 PM
• Nuria closed subtask T226104: Set up a generic workflow to create Kerberos accounts as Resolved.Aug 21 2019, 5:41 PM
• Nuria closed subtask T226089: Make the Kerberos infrastructure production ready as Resolved.Oct 24 2019, 7:08 PM
• Nuria closed subtask T226698: Allow all Analytics tools to work with Kerberos auth as Resolved.
• Nuria closed subtask T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster as Resolved.Nov 7 2019, 11:13 PM
• Nuria closed subtask T212258: Create test Kerberos identities/accounts for some selected users in hadoop test cluster as Resolved.Nov 22 2019, 11:03 PM
• Nuria closed subtask T237269: Prepare the Hadoop Analytics cluster for Kerberos as Resolved.Jan 7 2020, 2:00 PM
elukey closed subtask T235418: Add metadata to puppet about kerberos accounts as Resolved.Feb 4 2020, 11:07 AM
elukey closed this task as Resolved.Apr 17 2020, 1:33 PM

This monster can be finally close I believe :)

Ottomata subscribed.Apr 17 2020, 1:36 PM

🎉

elukey changed the status of subtask T227485: Decommission analytics10[28-31,33-41] from Stalled to Open.Sep 28 2020, 9:17 AM
• Cmjohnson closed subtask T227485: Decommission analytics10[28-31,33-41] as Resolved.Oct 7 2020, 2:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits