Page MenuHomePhabricator
Create Task
Maniphest T167790

Refactor puppet code for the Hadoop Analytics cluster to roles/profiles
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

While working on https://gerrit.wikimedia.org/r/#/c/354449 several comments were made to improve the hiera lookup logic in the Puppet Hadoop codebase.

This task should focus on:

  1. remove indirect hiera lookups, namely the ones happening not directly in a role but in another one that it is included. There are several FIXME comments in puppet after https://gerrit.wikimedia.org/r/#/c/354449, really easy to spot examples.
  1. move to profiles/roles, and possibly strive for having a single role for each host in site.pp.

Details

SubjectRepoBranchLines +/-
Refactor Hadoop code to allow more than one cluster in Prodoperations/puppetproduction+209 -164
Refactor hadoop/hive monitoring profiles to a simpler structureoperations/puppetproduction+73 -57
profile::hadoop:balancer: fix logrotate sourceoperations/puppetproduction+1 -1
Refactor the last bits of the Analytics code not following role/profileoperations/puppetproduction+21 -60
Refactor stat1005's roles into role/profilesoperations/puppetproduction+34 -30
role::analytics_cluster::client: move config to a separate profileoperations/puppetproduction+47 -40
Render role's analytics refinery logrotate from profileoperations/puppetproduction+1 -1
Move refinery::job::data_check from stat1005 to analytics1003operations/puppetproduction+1 -5
Move role refinery::job::* to profilesoperations/puppetproduction+244 -224
profile::hadoop::*: include labs firewall use caseoperations/puppetproduction+28 -14
role::analytics_cluster::hadoop::master: fix order in profile inclusionoperations/puppetproduction+1 -1
role::analytics_cluster: avoid to expicitly instance the standard classoperations/puppetproduction+3 -3
profile::hadoop: set hiera defaults to ease labs deploymentsoperations/puppetproduction+14 -8
Refactor thorium's roles in oneoperations/puppetproduction+113 -107
profile::hadoop::master: remove the last hadoop cdh auto-lookupoperations/puppetproduction+5 -2
role::analytics_cluster::coordinator: fix system::roleoperations/puppetproduction+1 -1
role::analytics_cluster::client: move to profilesoperations/puppetproduction+82 -64
role::analytics_cluster::[hadoop::standby|hue]: fix hiera configoperations/puppetproduction+7 -3
role::analytics_cluster::hive/oozie: move to profilesoperations/puppetproduction+84 -72
profile::hadoop::common: import hiera config from cdh::hadoopoperations/puppetproduction+289 -120
profile::hadoop::common,profile::hive::client: move hiera config in one placeoperations/puppetproduction+14 -78
Rename ::profile::hadoop::client to commons and move some features outoperations/puppetproduction+56 -52
profile::hadoop::client: move alarms definition to the worker profileoperations/puppetproduction+9 -12
profile::hadoop::worker: include the journalnode when neededoperations/puppetproduction+7 -1
Remove journalnode's config auto-inclusion from cdh::hadoopoperations/puppet/cdhmaster+3 -16
role::analytics_cluster::database::meta::backup*: move to profilesoperations/puppetproduction+52 -53
role::analytics_cluster::hadoop: move worker and masters to role/profilesoperations/puppetproduction+677 -496
role::analytics_cluster::hadoop::standby: restore namenode bacula backupoperations/puppetproduction+15 -10
Introduce role::analytics_cluster::coordinatoroperations/puppetproduction+64 -53
role::analytics_cluster::monitoring::disk: remove unused roleoperations/puppetproduction+0 -13
profile::druid::common: fix merge druid::propertiesoperations/puppetproduction+1 -1
role::analytics_cluster::hadoop::client: move to profiles (p2)operations/puppetproduction+130 -117
role::an_cluster::hadoop::client: moving to profiles (first part)operations/puppetproduction+27 -20
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
akosiaris subscribed.EditedJun 13 2017, 1:51 PM

Thanks for working on this!

  1. move to profiles/roles, and possibly strive for having a single role in site.pp.

I assume you mean a single role per host. That's the end goal after the adoption of the profile/role model for ops as well. It does maximize code re-use and easy configuration at the cost of an increased number of roles. But roles in the profile/role paradigm are so small and self-contained that that's fine.

• Nuria lowered the priority of this task from Medium to Low.Jun 15 2017, 3:53 PM
• Nuria moved this task from Incoming to Backlog (Later) on the Analytics board.
elukey updated the task description. (Show Details)Jun 19 2017, 8:34 AM
elukey added a project: User-Elukey.Jul 21 2017, 4:24 PM
gerritbot subscribed.Aug 4 2017, 12:38 PM

Change 370187 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::an_cluster::hadoop::client: moving to profiles (first part)

https://gerrit.wikimedia.org/r/370187

gerritbot added a project: Patch-For-Review.Aug 4 2017, 12:38 PM
elukey moved this task from Backlog to Stalled on the User-Elukey board.Aug 4 2017, 3:09 PM
elukey moved this task from Stalled to In Progress on the User-Elukey board.
elukey moved this task from In Progress to Analytics Backlog on the User-Elukey board.Aug 7 2017, 3:21 PM
gerritbot added a comment.Aug 9 2017, 12:43 PM

Change 370187 merged by Elukey:
[operations/puppet@production] role::an_cluster::hadoop::client: moving to profiles (first part)

https://gerrit.wikimedia.org/r/370187

gerritbot added a comment.Aug 9 2017, 1:09 PM

Change 370798 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hadoop::client: move to profiles (p2)

https://gerrit.wikimedia.org/r/370798

gerritbot added a comment.Aug 16 2017, 8:54 AM

Change 370798 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hadoop::client: move to profiles (p2)

https://gerrit.wikimedia.org/r/370798

gerritbot added a comment.Aug 16 2017, 9:50 AM

Change 372122 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::druid::common: fix merge druid::properties

https://gerrit.wikimedia.org/r/372122

gerritbot added a comment.Aug 16 2017, 9:53 AM

Change 372122 merged by Elukey:
[operations/puppet@production] profile::druid::common: fix merge druid::properties

https://gerrit.wikimedia.org/r/372122

gerritbot added a comment.Aug 16 2017, 10:40 AM

Change 372129 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::monitoring::disk: remove unused role

https://gerrit.wikimedia.org/r/372129

elukey edited projects, added Analytics-Kanban; removed Analytics.Aug 16 2017, 10:40 AM
gerritbot added a comment.Aug 16 2017, 10:41 AM

Change 372129 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::monitoring::disk: remove unused role

https://gerrit.wikimedia.org/r/372129

gerritbot added a comment.Aug 16 2017, 10:54 AM

Change 372131 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce role::analytics_cluster::coordinator

https://gerrit.wikimedia.org/r/372131

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Aug 16 2017, 3:00 PM
elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.Aug 28 2017, 11:54 AM
gerritbot added a comment.Sep 11 2017, 10:12 AM

Change 372131 merged by Elukey:
[operations/puppet@production] Introduce role::analytics_cluster::coordinator

https://gerrit.wikimedia.org/r/372131

elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Sep 22 2017, 3:02 PM
elukey moved this task from In Progress to Analytics Backlog on the User-Elukey board.Oct 5 2017, 8:27 AM
gerritbot added a comment.Nov 21 2017, 3:36 PM

Change 392658 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hadoop::worker: move to role/profiles

https://gerrit.wikimedia.org/r/392658

gerritbot added a comment.Nov 23 2017, 9:23 AM

Change 392658 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hadoop: move worker and masters to role/profiles

https://gerrit.wikimedia.org/r/392658

gerritbot added a comment.Nov 23 2017, 9:49 AM

Change 393041 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hadoop::standby: restore namenode bacula backup

https://gerrit.wikimedia.org/r/393041

gerritbot added a comment.Nov 23 2017, 9:59 AM

Change 393041 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hadoop::standby: restore namenode bacula backup

https://gerrit.wikimedia.org/r/393041

gerritbot added a comment.Nov 24 2017, 4:44 PM

Change 393257 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::database::meta::backup*: move to profiles

https://gerrit.wikimedia.org/r/393257

gerritbot added a comment.Nov 27 2017, 9:34 AM

Change 393257 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::database::meta::backup*: move to profiles

https://gerrit.wikimedia.org/r/393257

gerritbot added a comment.Nov 27 2017, 10:24 AM

Change 393563 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::client: move to profiles

https://gerrit.wikimedia.org/r/393563

gerritbot added a comment.Nov 27 2017, 3:59 PM

Change 393611 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet/cdh@master] Move journalnode's config from cdh::hadoop to cdh::hadoop:worker

https://gerrit.wikimedia.org/r/393611

gerritbot added a comment.Nov 27 2017, 7:10 PM

Change 393611 merged by Elukey:
[operations/puppet/cdh@master] Remove journalnode's config auto-inclusion from cdh::hadoop

https://gerrit.wikimedia.org/r/393611

gerritbot added a comment.Nov 27 2017, 7:14 PM

Change 393636 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::worker: include the journalnode when needed

https://gerrit.wikimedia.org/r/393636

gerritbot added a comment.Nov 27 2017, 7:21 PM

Change 393636 merged by Elukey:
[operations/puppet@production] profile::hadoop::worker: include the journalnode when needed

https://gerrit.wikimedia.org/r/393636

gerritbot added a comment.Nov 28 2017, 9:27 AM

Change 393733 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::client: move alarms definition to the worker profile

https://gerrit.wikimedia.org/r/393733

gerritbot added a comment.Nov 28 2017, 9:33 AM

Change 393733 merged by Elukey:
[operations/puppet@production] profile::hadoop::client: move alarms definition to the worker profile

https://gerrit.wikimedia.org/r/393733

elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.Nov 28 2017, 10:10 AM
gerritbot added a comment.Nov 28 2017, 10:12 AM

Change 393738 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Rename ::profile::hadoop::client to commons and move some features out

https://gerrit.wikimedia.org/r/393738

gerritbot added a comment.Nov 28 2017, 10:44 AM

Change 393741 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move Hadoop common profile hiera settings to common.yaml

https://gerrit.wikimedia.org/r/393741

gerritbot added a comment.Nov 28 2017, 12:47 PM

Change 393756 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::common: import hiera config from cdh::hadoop

https://gerrit.wikimedia.org/r/393756

gerritbot added a comment.Nov 29 2017, 8:50 AM

Change 393738 merged by Elukey:
[operations/puppet@production] Rename ::profile::hadoop::client to commons and move some features out

https://gerrit.wikimedia.org/r/393738

gerritbot added a comment.Nov 29 2017, 9:36 AM

Change 393741 merged by Elukey:
[operations/puppet@production] profile::hadoop::common,profile::hive::client: move hiera config in one place

https://gerrit.wikimedia.org/r/393741

gerritbot added a comment.Nov 29 2017, 4:20 PM

Change 393756 merged by Elukey:
[operations/puppet@production] profile::hadoop::common: import hiera config from cdh::hadoop

https://gerrit.wikimedia.org/r/393756

gerritbot added a comment.Dec 5 2017, 1:36 PM

Change 395527 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hive/oozie: move to profiles

https://gerrit.wikimedia.org/r/395527

gerritbot added a comment.Dec 6 2017, 10:03 AM

Change 395527 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hive/oozie: move to profiles

https://gerrit.wikimedia.org/r/395527

gerritbot added a comment.Dec 6 2017, 10:23 AM

Change 395710 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::[hadoop::standby|hue]: fix hiera config

https://gerrit.wikimedia.org/r/395710

gerritbot added a comment.Dec 6 2017, 10:29 AM

Change 395710 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::[hadoop::standby|hue]: fix hiera config

https://gerrit.wikimedia.org/r/395710

gerritbot added a comment.Dec 13 2017, 1:49 PM

Change 393563 abandoned by Elukey:
role::analytics_cluster::client: move to profiles

https://gerrit.wikimedia.org/r/393563

elukey moved this task from In Progress to Stalled on the User-Elukey board.Dec 18 2017, 10:45 AM
gerritbot added a comment.Jan 4 2018, 8:22 AM

Change 401904 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::master: remove the last hadoop cdh auto-lookup

https://gerrit.wikimedia.org/r/401904

gerritbot added a comment.Jan 4 2018, 8:28 AM

Change 401907 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::coordinator: fix system::role

https://gerrit.wikimedia.org/r/401907

gerritbot added a comment.Jan 4 2018, 8:29 AM

Change 401907 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::coordinator: fix system::role

https://gerrit.wikimedia.org/r/401907

gerritbot added a comment.Jan 4 2018, 10:09 AM

Change 401927 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Refactor thorium's roles in one

https://gerrit.wikimedia.org/r/401927

gerritbot added a comment.Jan 4 2018, 2:27 PM

Change 402050 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop: set hiera defaults to ease labs deployments

https://gerrit.wikimedia.org/r/402050

gerritbot added a comment.Jan 4 2018, 2:44 PM

Change 401904 merged by Elukey:
[operations/puppet@production] profile::hadoop::master: remove the last hadoop cdh auto-lookup

https://gerrit.wikimedia.org/r/401904

gerritbot added a comment.Jan 4 2018, 2:47 PM

Change 401927 merged by Elukey:
[operations/puppet@production] Refactor thorium's roles in one

https://gerrit.wikimedia.org/r/401927

gerritbot added a comment.Jan 4 2018, 3:56 PM

Change 402050 merged by Elukey:
[operations/puppet@production] profile::hadoop: set hiera defaults to ease labs deployments

https://gerrit.wikimedia.org/r/402050

gerritbot added a comment.Jan 4 2018, 5:26 PM

Change 402084 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster: avoid to expicitly instance the standard class

https://gerrit.wikimedia.org/r/402084

gerritbot added a comment.Jan 4 2018, 5:33 PM

Change 402084 merged by Elukey:
[operations/puppet@production] role::analytics_cluster: avoid to expicitly instance the standard class

https://gerrit.wikimedia.org/r/402084

• fdans moved this task from Paused to In Progress on the Analytics-Kanban board.Jan 4 2018, 5:41 PM
gerritbot added a comment.Jan 5 2018, 9:29 AM

Change 402316 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hadoop::master: fix order in profile inclusion

https://gerrit.wikimedia.org/r/402316

gerritbot added a comment.Jan 5 2018, 9:39 AM

Change 402316 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hadoop::master: fix order in profile inclusion

https://gerrit.wikimedia.org/r/402316

gerritbot added a comment.Jan 5 2018, 10:54 AM

Change 402323 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::*: include labs firewall use case

https://gerrit.wikimedia.org/r/402323

gerritbot added a comment.Jan 5 2018, 11:13 AM

Change 402323 abandoned by Elukey:
profile::hadoop::*: include labs firewall use case

https://gerrit.wikimedia.org/r/402323

gerritbot added a comment.Jan 8 2018, 3:32 PM

Change 402843 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move role refinery::job::* to profiles

https://gerrit.wikimedia.org/r/402843

gerritbot added a comment.Jan 8 2018, 3:43 PM

Change 402843 merged by Ottomata:
[operations/puppet@production] Move role refinery::job::* to profiles

https://gerrit.wikimedia.org/r/402843

gerritbot added a comment.Jan 8 2018, 3:53 PM

Change 402853 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move refinery::job::data_check from stat1005 to analytics1003

https://gerrit.wikimedia.org/r/402853

gerritbot added a comment.Jan 8 2018, 3:54 PM

Change 402853 merged by Ottomata:
[operations/puppet@production] Move refinery::job::data_check from stat1005 to analytics1003

https://gerrit.wikimedia.org/r/402853

gerritbot added a comment.Jan 8 2018, 3:58 PM

Change 402857 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Render role's analytics refinery logrotate from profile

https://gerrit.wikimedia.org/r/402857

gerritbot added a comment.Jan 8 2018, 4:00 PM

Change 402857 merged by Ottomata:
[operations/puppet@production] Render role's analytics refinery logrotate from profile

https://gerrit.wikimedia.org/r/402857

elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Feb 22 2018, 5:01 PM
gerritbot added a comment.Mar 19 2018, 10:51 AM

Change 420301 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::client: move config to a separate profile

https://gerrit.wikimedia.org/r/420301

gerritbot added a comment.Mar 19 2018, 2:17 PM

Change 420301 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::client: move config to a separate profile

https://gerrit.wikimedia.org/r/420301

gerritbot added a comment.Mar 19 2018, 5:36 PM

Change 420383 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Refactor stat1005's roles into role/profiles

https://gerrit.wikimedia.org/r/420383

gerritbot added a comment.Mar 20 2018, 6:44 AM

Change 420383 merged by Elukey:
[operations/puppet@production] Refactor stat1005's roles into role/profiles

https://gerrit.wikimedia.org/r/420383

gerritbot added a comment.Mar 20 2018, 8:03 AM

Change 420646 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Refactor the last bits of the Analytics code not following role/profile

https://gerrit.wikimedia.org/r/420646

elukey added a comment.Mar 20 2018, 9:16 AM

After the last puppet patch we should be in a good position regarding role/profiles. The last bit to discuss/review/refactor is profile::hadoop::common, since it takes its hiera values from a profile namespace, and overriding those values is not that easy (those ones are supposed to be global, so a new place in hiera might be better).

gerritbot added a comment.Mar 21 2018, 1:46 PM

Change 420646 merged by Elukey:
[operations/puppet@production] Refactor the last bits of the Analytics code not following role/profile

https://gerrit.wikimedia.org/r/420646

gerritbot added a comment.Mar 21 2018, 2:24 PM

Change 421020 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop:balancer: fix logrotate source

https://gerrit.wikimedia.org/r/421020

gerritbot added a comment.Mar 21 2018, 2:25 PM

Change 421020 merged by Elukey:
[operations/puppet@production] profile::hadoop:balancer: fix logrotate source

https://gerrit.wikimedia.org/r/421020

gerritbot added a comment.Mar 29 2018, 5:33 PM

Change 423000 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Refactor hadoop/hive monitoring profiles to a simpler structure

https://gerrit.wikimedia.org/r/423000

gerritbot added a comment.Mar 30 2018, 7:01 AM

Change 423000 merged by Elukey:
[operations/puppet@production] Refactor hadoop/hive monitoring profiles to a simpler structure

https://gerrit.wikimedia.org/r/423000

mforns edited projects, added Analytics; removed Analytics-Kanban.May 7 2018, 4:01 PM
mforns moved this task from Backlog (Later) to Incoming on the Analytics board.
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.May 14 2018, 3:54 PM
gerritbot added a comment.Jul 26 2018, 7:12 AM

Change 447813 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] [WIP] Refactor Hadoop code to allow more than one cluster in Prod

https://gerrit.wikimedia.org/r/447813

gerritbot added a comment.Aug 1 2018, 7:43 AM

Change 447813 merged by Elukey:
[operations/puppet@production] Refactor Hadoop code to allow more than one cluster in Prod

https://gerrit.wikimedia.org/r/447813

elukey moved this task from Stalled to Done on the User-Elukey board.Aug 1 2018, 9:10 AM
elukey added a comment.Aug 1 2018, 9:17 AM

The last patch ends the journey of this big refactoring, since we are now able to have multiple hadoop clusters defined in puppet (production). There might still be things here and there that needs to be ported to role/profile, but the bulk of the work is definitely done.

elukey edited projects, added Analytics-Kanban; removed Patch-For-Review.Aug 1 2018, 9:17 AM
elukey set the point value for this task to 21.
elukey moved this task from Paused to Done on the Analytics-Kanban board.
• Nuria closed this task as Resolved.Aug 21 2018, 3:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL