Page MenuHomePhabricator
Create Task
Maniphest T131974

Puppetize druid
Closed, ResolvedPublic34 Estimated Story Points
Actions

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+15 -6Druid puppet improvements for prod
operations/puppetproduction+7 -6Use ruby json lib to render Arrays as strings in druid runtime.properties.erb
operations/puppetproduction+4 -1Install the druid service package for each service
operations/puppetproduction+3 -2Add analytics_cluster::hadoop::client to druid workers so CDH is installed
operations/puppetproduction+80 -10Apply druid roles in production with initial (guesswork) configuration
operations/puppetproduction+1 K -0Druid module and analytics_cluster role class
operations/puppetproduction+1 -1Include cloudera reprepro updates in jessie-wikimedia
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT130258 Prototype Data Pipeline on Druid
 ResolvedOttomataT131974 Puppetize druid

Event Timeline

Nuria created this task.Apr 6 2016, 8:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2016, 8:51 PM
Nuria edited projects, added Analytics; removed Analytics-Kanban.Apr 6 2016, 8:51 PM
JAllemandou triaged this task as High priority.Apr 11 2016, 4:32 PM
JAllemandou moved this task from Incoming to Backlog (Later) on the Analytics board.
ori added a subscriber: ori.Apr 11 2016, 8:37 PM
Nuria added a parent task: T130258: Prototype Data Pipeline on Druid.May 5 2016, 5:14 PM
Nuria renamed this task from Productionitize druid to Puppetize druid.May 5 2016, 5:24 PM
Nuria updated the task description. (Show Details)
Nuria added a comment.May 5 2016, 5:32 PM

This can be done even if the absence of new hardware. How is the documentation?

This would take more or less time depending on the amount of processes that we have to set up and daemons that are running.

For example:

  • Will need to integrate with HDFS,
  • We will need to load data from druid into cassandra so it needs to talk to cassandra too. Is this just a query job? Or daemon configuration?

Do we need the hdfs labs cluster up to test this integration.

Nuria set the point value for this task to 34.May 5 2016, 5:33 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.May 10 2016, 4:02 PM
JAllemandou assigned this task to Ottomata.May 10 2016, 4:05 PM
gerritbot added a subscriber: gerritbot.May 10 2016, 9:19 PM

Change 288099 had a related patch set uploaded (by Ottomata):
[WIP] Druid module

https://gerrit.wikimedia.org/r/288099

gerritbot added a project: Patch-For-Review.May 10 2016, 9:19 PM
gerritbot added a comment.May 26 2016, 8:33 PM

Change 291043 had a related patch set uploaded (by Ottomata):
Include cloudera reprepro updates in jessie-wikimedia

https://gerrit.wikimedia.org/r/291043

gerritbot added a comment.May 26 2016, 8:36 PM

Change 291043 merged by Ottomata:
Include cloudera reprepro updates in jessie-wikimedia

https://gerrit.wikimedia.org/r/291043

Ottomata mentioned this in rOPUPafa4c55f79c6: Include cloudera reprepro updates in jessie-wikimedia.May 26 2016, 8:40 PM
gerritbot added a comment.May 26 2016, 8:49 PM

Change 288099 merged by Ottomata:
Druid module and analytics_cluster role class

https://gerrit.wikimedia.org/r/288099

Ottomata mentioned this in rOPUPb565a72519b3: Druid module and analytics_cluster role class.May 26 2016, 8:51 PM
gerritbot added a comment.May 26 2016, 10:03 PM

Change 291113 had a related patch set uploaded (by Ottomata):
Apply druid roles in production with initial (guesswork) configuration

https://gerrit.wikimedia.org/r/291113

gerritbot added a comment.May 26 2016, 10:13 PM

Change 291113 merged by Ottomata:
Apply druid roles in production with initial (guesswork) configuration

https://gerrit.wikimedia.org/r/291113

Ottomata mentioned this in rOPUPc2aca3160d7f: Apply druid roles in production with initial (guesswork) configuration.May 26 2016, 10:15 PM
gerritbot added a comment.May 26 2016, 10:42 PM

Change 291128 had a related patch set uploaded (by Ottomata):
Add analytics_cluster::hadoop::client to druid workers so CDH is installed

https://gerrit.wikimedia.org/r/291128

gerritbot added a comment.May 26 2016, 10:43 PM

Change 291128 merged by Ottomata:
Add analytics_cluster::hadoop::client to druid workers so CDH is installed

https://gerrit.wikimedia.org/r/291128

Ottomata mentioned this in rOPUP9b27af1435de: Add analytics_cluster::hadoop::client to druid workers so CDH is installed.May 26 2016, 10:47 PM
gerritbot added a comment.May 26 2016, 10:47 PM

Change 291129 had a related patch set uploaded (by Ottomata):
Install the druid service package for each service

https://gerrit.wikimedia.org/r/291129

gerritbot added a comment.May 26 2016, 10:48 PM

Change 291129 merged by Ottomata:
Install the druid service package for each service

https://gerrit.wikimedia.org/r/291129

Ottomata mentioned this in rOPUPe1d43a90a1bd: Install the druid service package for each service.May 26 2016, 10:51 PM
gerritbot added a comment.May 26 2016, 11:18 PM

Change 291137 had a related patch set uploaded (by Ottomata):
Use ruby json lib to render Arrays as strings in druid runtime.properties.erb

https://gerrit.wikimedia.org/r/291137

gerritbot added a comment.May 26 2016, 11:21 PM

Change 291137 merged by Ottomata:
Use ruby json lib to render Arrays as strings in druid runtime.properties.erb

https://gerrit.wikimedia.org/r/291137

Ottomata mentioned this in rOPUPc2d0eca8a809: Use ruby json lib to render Arrays as strings in druid runtime.properties.erb.May 26 2016, 11:23 PM
gerritbot added a comment.May 26 2016, 11:43 PM

Change 291140 had a related patch set uploaded (by Ottomata):
Druid puppet improvements for prod

https://gerrit.wikimedia.org/r/291140

gerritbot added a comment.May 26 2016, 11:44 PM

Change 291140 merged by Ottomata:
Druid puppet improvements for prod

https://gerrit.wikimedia.org/r/291140

Ottomata mentioned this in rOPUP3a8a6fc81658: Druid puppet improvements for prod.May 26 2016, 11:48 PM
Ottomata added a comment.May 27 2016, 1:47 PM

I'm calling this done!

There are going to be a lot of smaller follow up tasks, especially for monitoring.

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.May 27 2016, 1:47 PM
Nuria closed this task as Resolved.May 30 2016, 9:55 PM
Ottomata mentioned this in rOPUP39e2eb1a461d: Add analytics_cluster::hadoop::client to druid workers so CDH is installed.Jun 17 2016, 6:07 PM
Ottomata mentioned this in rOPUP899f1fe3f7df: Apply druid roles in production with initial (guesswork) configuration.
Ottomata mentioned this in rOPUP722ff65da5cd: Apply druid roles in production with initial (guesswork) configuration.
Ottomata mentioned this in rOPUP5c28d8cab71b: Apply druid roles in production with initial (guesswork) configuration.
Ottomata mentioned this in rOPUPdada1faff7ef: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP5dc9f98dcce8: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUPb18cf670a77b: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUPb6f3b0eaf2f2: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP67dfa96d5f8a: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP0de8c05932ff: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP8345a2878ce9: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP51c3b447e98d: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP12ba54b0c623: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUPe8f192d3d3d0: Druid module and analytics_cluster role class.
Ottomata mentioned this in rOPUP38bfb2810b6d: Druid module.
Ottomata mentioned this in rOPUP22d749400f8e: Druid module.
Ottomata mentioned this in rOPUP673af81ca549: Druid module.
Ottomata mentioned this in rOPUP8c00528aecdb: [WIP] Druid module.
Ottomata mentioned this in rOPUP160e933be345: [WIP] Druid module.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL