Page MenuHomePhabricator
Create Task
Maniphest T212259

Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings.
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings.

Details

SubjectRepoBranchLines +/-
Add oozie webrequest test bundleanalytics/refinerymaster+246 -0
role::analytics_test_cluster::hadoop::ui: configure hive for hueoperations/puppetproduction+5 -0
cdh::oozie: add hive2/hcat credentials classesoperations/puppetproduction+8 -0
profile::hadoop::common: set r+o to the trustore fileoperations/puppetproduction+1 -1
camus: add support for kerberosoperations/puppetproduction+21 -13
Replace profile::analytics::systemd_timer with kerberos::systemd_timeroperations/puppetproduction+224 -165
Set oozie as proxy for the Hadoop testing clusteroperations/puppetproduction+2 -0
Introduce the kerberos moduleoperations/puppetproduction+39 -31
Add cdh::systemd_timeroperations/puppet/cdhmaster+122 -0
hadoop: format dfs.cluster.administrators correctlyoperations/puppetproduction+1 -1
hadoop: set 'hdfs' as admin user for the Hadoop test clusteroperations/puppetproduction+1 -1
role::analytics_test_cluste::hadoop::master|stanby: allow https portoperations/puppetproduction+2 -0
profile::hue: add a parameter to selectively enable oozie securityoperations/puppetproduction+4 -1
Move ensure hive-site.xml from (test) hadoo coord to uioperations/puppetproduction+3 -3
role::analytics_test_cluster::coordinator: ensure hive-site.xml in HDFSoperations/puppetproduction+3 -1
profile::analytics::refinery::job::test::camus: fix checked topicoperations/puppetproduction+1 -1
profile::analytics::refinery::job::test::camus: fix topic whitelistoperations/puppetproduction+1 -1
Rename kafka webrequest test topicoperations/puppetproduction+2 -2
camus: make webrequest_text config more similar to prodoperations/puppetproduction+2 -2
role::analytics_test_cluster::coord: add kafkatee instanceoperations/puppetproduction+50 -0
role::analytics_test_cluster::coordinator: add admin settingsoperations/puppetproduction+10 -4
role::analytics_test_cluster::coordinator: add basic camus supportoperations/puppetproduction+238 -7
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• Nuria triaged this task as High priority.Dec 18 2018, 9:33 PM
• Nuria created this task.
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.Jan 7 2019, 4:37 PM
elukey mentioned this in T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..Jan 22 2019, 11:26 AM
elukey added a project: Analytics-Kanban.
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Feb 8 2019, 4:00 PM

Change 489243 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordinator: add basic camus support

https://gerrit.wikimedia.org/r/489243

gerritbot added a project: Patch-For-Review.Feb 8 2019, 4:00 PM
gerritbot added a comment.Feb 12 2019, 9:36 AM

Change 489243 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordinator: add basic camus support

https://gerrit.wikimedia.org/r/489243

gerritbot added a comment.Feb 12 2019, 10:06 AM

Change 490004 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordinator: add admin settings

https://gerrit.wikimedia.org/r/490004

gerritbot added a comment.Feb 12 2019, 10:09 AM

Change 490004 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordinator: add admin settings

https://gerrit.wikimedia.org/r/490004

gerritbot added a comment.Feb 12 2019, 2:41 PM

Change 490067 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coord: add kafkatee instance

https://gerrit.wikimedia.org/r/490067

gerritbot added a comment.Feb 13 2019, 4:43 PM

Change 490067 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coord: add kafkatee instance

https://gerrit.wikimedia.org/r/490067

gerritbot added a comment.Feb 20 2019, 3:25 PM

Change 491777 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] camus: make webrequest_text config more similar to prod

https://gerrit.wikimedia.org/r/491777

gerritbot added a comment.Feb 20 2019, 3:29 PM

Change 491777 merged by Elukey:
[operations/puppet@production] camus: make webrequest_text config more similar to prod

https://gerrit.wikimedia.org/r/491777

gerritbot added a comment.Feb 20 2019, 3:40 PM

Change 491779 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Rename kafka webrequest test topic

https://gerrit.wikimedia.org/r/491779

gerritbot added a comment.Feb 20 2019, 3:44 PM

Change 491779 merged by Elukey:
[operations/puppet@production] Rename kafka webrequest test topic

https://gerrit.wikimedia.org/r/491779

elukey added a comment.Feb 20 2019, 4:07 PM
hdfs@analytics1030:/mnt/hdfs/wmf/data/raw/webrequest$ ls
webrequest_test_text

So if I got it correctly, it is now only a matter of using the right bundle.xml/properties and we should be able to refine this data too!

gerritbot added a comment.Feb 20 2019, 4:20 PM

Change 491791 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery@master] Add oozie webrequest test bundle

https://gerrit.wikimedia.org/r/491791

gerritbot added a comment.Feb 21 2019, 11:21 AM

Change 491944 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::refinery::job::test::camus: fix topic whitelist

https://gerrit.wikimedia.org/r/491944

gerritbot added a comment.Feb 21 2019, 11:22 AM

Change 491944 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::test::camus: fix topic whitelist

https://gerrit.wikimedia.org/r/491944

gerritbot added a comment.Feb 21 2019, 12:00 PM

Change 491949 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::refinery::job::test::camus: fix checked topic

https://gerrit.wikimedia.org/r/491949

gerritbot added a comment.Feb 21 2019, 12:02 PM

Change 491949 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::test::camus: fix checked topic

https://gerrit.wikimedia.org/r/491949

gerritbot added a comment.Feb 21 2019, 2:35 PM

Change 491969 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordinator: ensure hive-site.xml in HDFS

https://gerrit.wikimedia.org/r/491969

gerritbot added a comment.Feb 21 2019, 2:37 PM

Change 491969 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordinator: ensure hive-site.xml in HDFS

https://gerrit.wikimedia.org/r/491969

gerritbot added a comment.Feb 21 2019, 2:44 PM

Change 491973 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move ensure hive-site.xml from (test) hadoo coord to ui

https://gerrit.wikimedia.org/r/491973

gerritbot added a comment.Feb 21 2019, 2:46 PM

Change 491973 merged by Elukey:
[operations/puppet@production] Move ensure hive-site.xml from (test) hadoo coord to ui

https://gerrit.wikimedia.org/r/491973

Ottomata subscribed.Mar 26 2019, 8:08 PM

Just saw this on the Confluent mailing list. Parking it here for future reference:

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/confluent-platform/twFcx3H689U/x9g3WANKBAAJ

We don't yet use Kafka Connect, but if/when we do and we have a Kerberized Hadoop Cluster, we'll need to be aware of that. :/

elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Mar 28 2019, 9:16 AM
elukey moved this task from Paused to In Progress on the Analytics-Kanban board.Apr 12 2019, 9:04 AM
elukey added a project: User-Elukey.Apr 15 2019, 12:55 PM
elukey moved this task from Backlog to In Progress on the User-Elukey board.Apr 16 2019, 11:02 AM
elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Jun 4 2019, 4:03 PM
elukey moved this task from In Progress to Stalled on the User-Elukey board.Jun 6 2019, 4:21 PM
elukey moved this task from Paused to In Progress on the Analytics-Kanban board.Jun 20 2019, 2:11 PM
gerritbot added a comment.Jun 20 2019, 6:03 PM

Change 518097 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet/cdh@master] Add cdh::systemd_timer

https://gerrit.wikimedia.org/r/518097

gerritbot added a comment.Jun 21 2019, 8:28 AM

Change 518220 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hue: add a parameter to selectively enable oozie security

https://gerrit.wikimedia.org/r/518220

gerritbot added a comment.Jun 21 2019, 8:41 AM

Change 518220 merged by Elukey:
[operations/puppet@production] profile::hue: add a parameter to selectively enable oozie security

https://gerrit.wikimedia.org/r/518220

gerritbot added a comment.Jun 24 2019, 6:23 AM

Change 518469 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluste::hadoop::master|stanby: allow https port

https://gerrit.wikimedia.org/r/518469

gerritbot added a comment.Jun 24 2019, 6:23 AM

Change 518469 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluste::hadoop::master|stanby: allow https port

https://gerrit.wikimedia.org/r/518469

gerritbot added a comment.Jun 24 2019, 7:10 AM

Change 518646 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hadoop: set 'hdfs' as admin user for the Hadoop test cluster

https://gerrit.wikimedia.org/r/518646

gerritbot added a comment.Jun 24 2019, 7:13 AM

Change 518646 merged by Elukey:
[operations/puppet@production] hadoop: set 'hdfs' as admin user for the Hadoop test cluster

https://gerrit.wikimedia.org/r/518646

gerritbot added a comment.Jun 24 2019, 7:41 AM

Change 518648 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hadoop: format dfs.cluster.administrators correctly

https://gerrit.wikimedia.org/r/518648

gerritbot added a comment.Jun 24 2019, 7:42 AM

Change 518648 merged by Elukey:
[operations/puppet@production] hadoop: format dfs.cluster.administrators correctly

https://gerrit.wikimedia.org/r/518648

• Nuria closed subtask Restricted Task as Resolved.Jun 24 2019, 9:11 PM
gerritbot added a comment.Jun 25 2019, 6:53 AM

Change 518097 abandoned by Elukey:
Add cdh::systemd_timer

https://gerrit.wikimedia.org/r/518097

gerritbot added a comment.Jun 25 2019, 8:35 AM

Change 518915 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce the kerberos module

https://gerrit.wikimedia.org/r/518915

gerritbot added a comment.Jun 25 2019, 9:17 AM

Change 518915 merged by Elukey:
[operations/puppet@production] Introduce the kerberos module

https://gerrit.wikimedia.org/r/518915

gerritbot added a comment.Jun 25 2019, 10:36 AM

Change 518954 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Replace profile::analytics::systemd_timer with kerberos::systemd_timer

https://gerrit.wikimedia.org/r/518954

gerritbot added a comment.Jun 25 2019, 10:53 AM

Change 518958 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] camus: add support for kerberos

https://gerrit.wikimedia.org/r/518958

gerritbot added a comment.Jun 25 2019, 3:53 PM

Change 519057 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set oozie as proxy for the Hadoop testing cluster

https://gerrit.wikimedia.org/r/519057

gerritbot added a comment.Jun 25 2019, 3:54 PM

Change 519057 merged by Elukey:
[operations/puppet@production] Set oozie as proxy for the Hadoop testing cluster

https://gerrit.wikimedia.org/r/519057

gerritbot added a comment.Jun 26 2019, 1:27 PM

Change 518954 merged by Elukey:
[operations/puppet@production] Replace profile::analytics::systemd_timer with kerberos::systemd_timer

https://gerrit.wikimedia.org/r/518954

gerritbot added a comment.Jun 26 2019, 1:31 PM

Change 518958 merged by Elukey:
[operations/puppet@production] camus: add support for kerberos

https://gerrit.wikimedia.org/r/518958

gerritbot added a comment.Jun 26 2019, 5:27 PM

Change 519263 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::common: set r+o to the trustore file

https://gerrit.wikimedia.org/r/519263

gerritbot added a comment.Jun 26 2019, 5:28 PM

Change 519263 merged by Elukey:
[operations/puppet@production] profile::hadoop::common: set r+o to the trustore file

https://gerrit.wikimedia.org/r/519263

elukey added a comment.Jun 27 2019, 7:30 AM

Summary of things done:

  1. hue works now with oozie
  2. oozie is kerberized
  3. camus works fine with kerberos
  4. webrequest_load runs successfully replacing hive actions with hive2 actions

Point 4) example:

diff --git a/oozie/util/hive/partition/add/workflow.xml b/oozie/util/hive/partition/add/workflow.xml
index ca07199..4e6d892 100644
--- a/oozie/util/hive/partition/add/workflow.xml
+++ b/oozie/util/hive/partition/add/workflow.xml
@@ -42,13 +42,26 @@
             <description>HDFS path(s) naming the input dataset.</description>
         </property>
     </parameters>
+      <credentials>
+         <credential name='my-hive-creds' type='hive2'>
+            <property>
+               <name>hive2.server.principal</name>
+               <value>hive/analytics1030.eqiad.wmnet@WIKIMEDIA</value>
+            </property>
+            <property>
+               <name>hive2.jdbc.url</name>
+               <value>jdbc:hive2://analytics1030.eqiad.wmnet:10000/default</value>
+            </property>
+         </credential>
+      </credentials>
+

     <start to="add_partition"/>

-    <action name="add_partition">
-        <hive xmlns="uri:oozie:hive-action:0.2">
+    <action name="add_partition" cred="my-hive-creds">
+        <hive2 xmlns="uri:oozie:hive2-action:0.1">
             <job-tracker>${job_tracker}</job-tracker>
             <name-node>${name_node}</name-node>
             <job-xml>${hive_site_xml}</job-xml>
             <configuration>
                 <!--make sure oozie:launcher runs in a low priority queue -->
@@ -65,13 +78,13 @@
                     <value>${queue_name}</value>
                 </property>
             </configuration>
-
+            <jdbc-url>jdbc:hive2://analytics1030.eqiad.wmnet:10000/default</jdbc-url>
             <script>${hive_script}</script>
             <param>database=${replaceAll(table, "\\..*", "")}</param>
             <param>table=${replaceAll(table, "^.*\\.", "")}</param>
             <param>location=${location}</param>
             <param>partition_spec=${partition_spec}</param>
-        </hive>
+        </hive2>
         <ok to="end"/>
         <error to="kill"/>
     </action>
gerritbot added a comment.Jun 27 2019, 7:50 AM

Change 519355 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cdh::oozie: add hive2/hcat credentials classes

https://gerrit.wikimedia.org/r/519355

gerritbot added a comment.Jun 27 2019, 7:51 AM

Change 519355 merged by Elukey:
[operations/puppet@production] cdh::oozie: add hive2/hcat credentials classes

https://gerrit.wikimedia.org/r/519355

elukey added a comment.Jun 27 2019, 9:01 AM

For the scope of this task, I would call it done. Several follow ups will need to be done, but overall most of the critical analytics tools are working!

elukey set the point value for this task to 13.Jun 27 2019, 9:01 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
elukey mentioned this in T226698: Allow all Analytics tools to work with Kerberos auth.Jun 27 2019, 9:41 AM
gerritbot added a comment.Jun 27 2019, 10:23 AM

Change 519368 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::hadoop::ui: configure hive for hue

https://gerrit.wikimedia.org/r/519368

gerritbot added a comment.Jun 27 2019, 10:24 AM

Change 519368 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::hadoop::ui: configure hive for hue

https://gerrit.wikimedia.org/r/519368

• Nuria added a comment.Jun 27 2019, 4:47 PM

Nice, major milestone.

• Nuria closed this task as Resolved.Jun 27 2019, 4:47 PM
• fdans closed subtask T226232: Update the Camus checker to be able to authenticate via Kerberos as Resolved.Jul 1 2019, 3:50 PM
elukey mentioned this in T227257: Move refinery to hive 2 actions.Jul 4 2019, 8:12 AM
gerritbot added a comment.Oct 23 2020, 8:32 AM

Change 491791 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery@master] Add oozie webrequest test bundle

https://gerrit.wikimedia.org/r/491791

gerritbot added a comment.May 12 2022, 9:11 AM

Change 491791 abandoned by Elukey:

[analytics/refinery@master] Add oozie webrequest test bundle

Reason:

https://gerrit.wikimedia.org/r/491791

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL