Page MenuHomePhabricator
Create Task
Maniphest T109859

Refactor analytics/cdh roles to use hiera, setup Analytics Cluster in beta labs. [21 pts]
Closed, ResolvedPublic
Actions

Description

It is about time we created and maintained an official Analytics cluster in Beta. This includes CDH (Hadoop) and Kafka. It might also include varnishkafka on Beta varnishes.

There will be a little bit of refactoring in puppet that needs to happen for this, but for the most part the existing puppetization will do everything it needs to do. Building this cluster won't be too much work (maybe half a week), but maintaining it might be.

This work does not have to be done by me. I will work on any puppet tweaks needed, but I think it would be good to get more analytics engineers involved here.

Details

SubjectRepoBranchLines +/-
Use project-$labsproject in labs for /var/log/refineryoperations/puppetproduction+6 -2
Configure Analytics Cluster in beta deployment-prepoperations/puppetproduction+16 -0
Remove unused anayltics role from analytics kafka brokersoperations/puppetproduction+0 -1
Remove unused analytics role classesoperations/puppetproduction+2 -1 K
Remove some unsed role::analytics::* classes, more to comeoperations/puppetproduction+7 -78
Apply new analytics_cluster role to analytics1027operations/puppetproduction+62 -44
Create refinery classes in analytics_cluster role, apply them to stat1002operations/puppetproduction+298 -40
Use new analytics_cluster::hadoop::worker role on all Hadoop workersoperations/puppetproduction+4 -9
Use new analytics_cluster::hadoop::master role on analytics1001operations/puppetproduction+2 -2
Move hadoop_user_posix_groups setting to eqiad/cdh/hadoop/users.yamloperations/puppetproduction+12 -20
Apply new analytics_cluster::hadoop::standby role on analytics1002operations/puppetproduction+1 -1
Include new analytics_cluster::hadoop::worker for testing on analytics1057operations/puppetproduction+6 -1
Include analytics_cluster::client role on analytics1026 for testingoperations/puppetproduction+1 -1
Move hadoop memory settings for production above class declarationoperations/puppetproduction+106 -104
Fix use of variable in analytics_cluster/hadoop/client.ppoperations/puppetproduction+1 -1
Add missing net-topology.py.erb to analytics_cluster roleoperations/puppetproduction+66 -0
Move analytics_cluster prod hiera vars to eqiad/cdh/operations/puppetproduction+64 -52
Add role::analytics_cluster::java classoperations/puppetproduction+13 -0
Add hive and oozie database grant definesoperations/puppet/cdhmaster+76 -0
Rename analytics_new to analytics_clusteroperations/puppetproduction+99 -99
Refactor manifests/role/analytics/* into modules/role, use hiera to configureoperations/puppetproduction+829 -2
Updates to work with hiera, Hive/Oozie MySQL db can now be hosted on remote nodeoperations/puppet/cdhmaster+124 -138
Show related patches Customize query in gerrit

Event Timeline

Ottomata created this task.Aug 21 2015, 5:17 PM
Ottomata raised the priority of this task from to Needs Triage.
Ottomata updated the task description. (Show Details)
Ottomata added projects: Analytics-Backlog, Analytics-Clusters.
Ottomata added subscribers: Ottomata, • kevinator, Milimetric and 3 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2015, 5:17 PM
Milimetric triaged this task as Low priority.Aug 27 2015, 5:15 PM
Milimetric set Security to None.
Milimetric moved this task from Incoming to Prioritized on the Analytics-Backlog board.
Milimetric moved this task from Prioritized to Backlog on the Analytics-Backlog board.Oct 15 2015, 5:18 PM
• ggellerman edited projects, added Analytics; removed Analytics-Backlog.Jan 12 2016, 7:41 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.Jan 12 2016, 7:41 PM
Ottomata added a subscriber: elukey.Jan 22 2016, 3:56 PM

@elukey woo woooo

JAllemandou added a comment.Jan 22 2016, 4:59 PM

@elukey I'll ensure you a beer supplied night a next offsite for that one :)

Ottomata edited projects, added Analytics-Kanban; removed Analytics.Jan 29 2016, 7:00 PM
gerritbot subscribed.Feb 1 2016, 11:22 PM

Change 267797 had a related patch set uploaded (by Ottomata):
[WIP] Refactor manifests/role/analytics/* into modules/role, use hiera to configure

https://gerrit.wikimedia.org/r/267797

gerritbot added a project: Patch-For-Review.Feb 1 2016, 11:22 PM
Ottomata claimed this task.Feb 3 2016, 5:01 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
Ottomata renamed this task from Create and maintain an Analytics Cluster in Beta Cluster in labs. to Create and maintain an Analytics Cluster in Beta Cluster in labs. [21 pts].Feb 4 2016, 6:03 PM
Ottomata renamed this task from Create and maintain an Analytics Cluster in Beta Cluster in labs. [21 pts] to Refactor analytics/cdh roles to use hiera, setup Analytics Cluster in beta labs. [21 pts].Feb 8 2016, 11:37 PM
gerritbot added a comment.Feb 9 2016, 12:17 AM

Change 269340 had a related patch set uploaded (by Ottomata):
Updates to work with hiera, Hive/Oozie MySQL db can now be hosted on remote node

https://gerrit.wikimedia.org/r/269340

gerritbot added a comment.Feb 9 2016, 8:11 PM

Change 269340 merged by Ottomata:
Updates to work with hiera, Hive/Oozie MySQL db can now be hosted on remote node

https://gerrit.wikimedia.org/r/269340

gerritbot added a comment.Feb 9 2016, 8:32 PM

Change 267797 merged by Ottomata:
Refactor manifests/role/analytics/* into modules/role, use hiera to configure

https://gerrit.wikimedia.org/r/267797

gerritbot added a comment.Feb 10 2016, 8:57 PM

Change 269825 had a related patch set uploaded (by Ottomata):
Rename analytics_new to analytics_cluster

https://gerrit.wikimedia.org/r/269825

gerritbot added a comment.Feb 10 2016, 9:02 PM

Change 269825 merged by Ottomata:
Rename analytics_new to analytics_cluster

https://gerrit.wikimedia.org/r/269825

gerritbot added a comment.Feb 10 2016, 9:47 PM

Change 269841 had a related patch set uploaded (by Ottomata):
Add hive and oozie database grant defines

https://gerrit.wikimedia.org/r/269841

gerritbot added a comment.Feb 10 2016, 9:48 PM

Change 269841 merged by Ottomata:
Add hive and oozie database grant defines

https://gerrit.wikimedia.org/r/269841

gerritbot added a comment.Feb 10 2016, 9:57 PM

Change 269844 had a related patch set uploaded (by Ottomata):
Include analytics_cluster::client role on analytics1026 for testing

https://gerrit.wikimedia.org/r/269844

gerritbot added a comment.Feb 11 2016, 3:24 PM

Change 269981 had a related patch set uploaded (by Ottomata):
Add role::analytics_cluster::java class

https://gerrit.wikimedia.org/r/269981

gerritbot added a comment.Feb 11 2016, 3:24 PM

Change 269981 merged by Ottomata:
Add role::analytics_cluster::java class

https://gerrit.wikimedia.org/r/269981

gerritbot added a comment.Feb 11 2016, 4:15 PM

Change 269991 had a related patch set uploaded (by Ottomata):
Move analytics_cluster prod hiera vars to eqiad/cdh/

https://gerrit.wikimedia.org/r/269991

gerritbot added a comment.Feb 11 2016, 4:19 PM

Change 269991 merged by Ottomata:
Move analytics_cluster prod hiera vars to eqiad/cdh/

https://gerrit.wikimedia.org/r/269991

gerritbot added a comment.Feb 11 2016, 4:29 PM

Change 269994 had a related patch set uploaded (by Ottomata):
Add missing net-topology.py.erb to analytics_cluster role

https://gerrit.wikimedia.org/r/269994

gerritbot added a comment.Feb 11 2016, 4:34 PM

Change 269994 merged by Ottomata:
Add missing net-topology.py.erb to analytics_cluster role

https://gerrit.wikimedia.org/r/269994

gerritbot added a comment.Feb 11 2016, 4:38 PM

Change 269996 had a related patch set uploaded (by Ottomata):
Fix use of variable in analytics_cluster/hadoop/client.pp

https://gerrit.wikimedia.org/r/269996

gerritbot added a comment.Feb 11 2016, 4:41 PM

Change 269996 merged by Ottomata:
Fix use of variable in analytics_cluster/hadoop/client.pp

https://gerrit.wikimedia.org/r/269996

gerritbot added a comment.Feb 11 2016, 4:46 PM

Change 269998 had a related patch set uploaded (by Ottomata):
Move hadoop memory settings for production above class declaration

https://gerrit.wikimedia.org/r/269998

gerritbot added a comment.Feb 11 2016, 4:48 PM

Change 269998 merged by Ottomata:
Move hadoop memory settings for production above class declaration

https://gerrit.wikimedia.org/r/269998

gerritbot added a comment.Feb 11 2016, 7:34 PM

Change 269844 merged by Ottomata:
Include analytics_cluster::client role on analytics1026 for testing

https://gerrit.wikimedia.org/r/269844

gerritbot added a comment.Feb 11 2016, 7:44 PM

Change 270033 had a related patch set uploaded (by Ottomata):
Include new analytics_cluster::hadoop::worker for testing on analytics1057

https://gerrit.wikimedia.org/r/270033

gerritbot added a comment.Feb 11 2016, 7:47 PM

Change 270033 merged by Ottomata:
Include new analytics_cluster::hadoop::worker for testing on analytics1057

https://gerrit.wikimedia.org/r/270033

gerritbot added a comment.Feb 11 2016, 7:53 PM

Change 270035 had a related patch set uploaded (by Ottomata):
Apply new analytics_cluster::hadoop::standby role on analytics1002

https://gerrit.wikimedia.org/r/270035

gerritbot added a comment.Feb 11 2016, 7:57 PM

Change 270035 merged by Ottomata:
Apply new analytics_cluster::hadoop::standby role on analytics1002

https://gerrit.wikimedia.org/r/270035

gerritbot added a comment.Feb 11 2016, 7:58 PM

Change 270037 had a related patch set uploaded (by Ottomata):
Use new analytics_cluster::hadoop::master role on analytics1001

https://gerrit.wikimedia.org/r/270037

gerritbot added a comment.Feb 11 2016, 8:14 PM

Change 270040 had a related patch set uploaded (by Ottomata):
Move hadoop_user_posix_groups setting to eqiad/cdh/hadoop/users.yaml

https://gerrit.wikimedia.org/r/270040

gerritbot added a comment.Feb 11 2016, 8:18 PM

Change 270040 merged by Ottomata:
Move hadoop_user_posix_groups setting to eqiad/cdh/hadoop/users.yaml

https://gerrit.wikimedia.org/r/270040

gerritbot added a comment.Feb 11 2016, 8:23 PM

Change 270037 merged by Ottomata:
Use new analytics_cluster::hadoop::master role on analytics1001

https://gerrit.wikimedia.org/r/270037

gerritbot added a comment.Feb 11 2016, 8:37 PM

Change 270050 had a related patch set uploaded (by Ottomata):
Use new analytics_cluster::hadoop::worker role on all Hadoop workers

https://gerrit.wikimedia.org/r/270050

gerritbot added a comment.Feb 11 2016, 8:40 PM

Change 270050 merged by Ottomata:
Use new analytics_cluster::hadoop::worker role on all Hadoop workers

https://gerrit.wikimedia.org/r/270050

gerritbot added a comment.Feb 11 2016, 8:59 PM

Change 270103 had a related patch set uploaded (by Ottomata):
Create refinery classes in analytics_cluster role, apply them to stat1002

https://gerrit.wikimedia.org/r/270103

gerritbot added a comment.Feb 15 2016, 7:32 PM

Change 270103 merged by Ottomata:
Create refinery classes in analytics_cluster role, apply them to stat1002

https://gerrit.wikimedia.org/r/270103

gerritbot added a comment.Feb 15 2016, 7:43 PM

Change 270795 had a related patch set uploaded (by Ottomata):
Apply new analytics_cluster role to analytics1027

https://gerrit.wikimedia.org/r/270795

gerritbot added a comment.Feb 15 2016, 8:25 PM

Change 270795 merged by Ottomata:
Apply new analytics_cluster role to analytics1027

https://gerrit.wikimedia.org/r/270795

gerritbot added a comment.Feb 15 2016, 8:35 PM

Change 270808 had a related patch set uploaded (by Ottomata):
Remove some unsed role::analytics::* classes, more to come

https://gerrit.wikimedia.org/r/270808

gerritbot added a comment.Feb 15 2016, 8:38 PM

Change 270808 merged by Ottomata:
Remove some unsed role::analytics::* classes, more to come

https://gerrit.wikimedia.org/r/270808

gerritbot added a comment.Feb 15 2016, 8:49 PM

Change 270851 had a related patch set uploaded (by Ottomata):
Remove unused analytics role classes

https://gerrit.wikimedia.org/r/270851

gerritbot added a comment.Feb 15 2016, 8:54 PM

Change 270851 merged by Ottomata:
Remove unused analytics role classes

https://gerrit.wikimedia.org/r/270851

gerritbot added a comment.Feb 15 2016, 9:10 PM

Change 270857 had a related patch set uploaded (by Ottomata):
Remove unused anayltics role from analytics kafka brokers

https://gerrit.wikimedia.org/r/270857

gerritbot added a comment.Feb 15 2016, 9:11 PM

Change 270857 merged by Ottomata:
Remove unused anayltics role from analytics kafka brokers

https://gerrit.wikimedia.org/r/270857

gerritbot added a comment.Feb 16 2016, 7:53 PM

Change 271044 had a related patch set uploaded (by Ottomata):
Configure Analytics Cluster in beta deployment-prep

https://gerrit.wikimedia.org/r/271044

gerritbot added a comment.Feb 16 2016, 7:57 PM

Change 271044 merged by Ottomata:
Configure Analytics Cluster in beta deployment-prep

https://gerrit.wikimedia.org/r/271044

gerritbot added a comment.Feb 16 2016, 8:44 PM

Change 271062 had a related patch set uploaded (by Ottomata):
Use project-$labsproject in labs for /var/log/refinery

https://gerrit.wikimedia.org/r/271062

gerritbot added a comment.Feb 16 2016, 8:48 PM

Change 271062 merged by Ottomata:
Use project-$labsproject in labs for /var/log/refinery

https://gerrit.wikimedia.org/r/271062

Ottomata added a comment.Feb 16 2016, 9:13 PM

Yeehaw! Done!

deployment-analytics03 should be our main entrypoint for using hadoop clients (Hive, Spark, etc.)

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Feb 16 2016, 9:13 PM
• Nuria closed this task as Resolved.Feb 18 2016, 4:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL