Page MenuHomePhabricator
Create Task
Maniphest T188377

Import some Analytics git puppet submodules to operations/puppet
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

The following submodules are good candidates to be imported in operations/puppet:

[submodule "modules/zookeeper"]
	path = modules/zookeeper
	url = https://gerrit.wikimedia.org/r/operations/puppet/zookeeper
[submodule "modules/jmxtrans"]
	path = modules/jmxtrans
	url = https://gerrit.wikimedia.org/r/operations/puppet/jmxtrans
[submodule "modules/varnishkafka"]
	path = modules/varnishkafka
	url = https://gerrit.wikimedia.org/r/operations/puppet/varnishkafka
[submodule "modules/kafkatee"]
	path = modules/kafkatee
	url = https://gerrit.wikimedia.org/r/operations/puppet/kafkatee

Given the few contributions during these years it would make sense to import them into operations/puppet in order to ease their maintenance. It also happened in the past that git submodule sha updates in operations/puppet were rolled-out/reverted accidentally by people making unrelated changes to the operations/puppet repo, risking outages that could have been avoided.

Details

SubjectRepoBranchLines +/-
Move the nginx submodule to operations/puppetoperations/puppetproduction+926 -4
Move the varnishkafka submodule to operations/puppetoperations/puppetproduction+1 K -4
Move the kafkatee submodule to operations/puppetoperations/puppetproduction+489 -4
Move the jmxtrans submodule to operations/puppetoperations/puppetproduction+465 -4
Remove puppet submodules merged into operations/puppetintegration/configmaster+0 -14
Move away jmxtrans/kafkatee's modules from environments/productionoperations/puppetproduction+0 -0
Move the nginx submodule to operations/puppetoperations/puppetproduction+0 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Feb 27 2018, 8:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 27 2018, 8:05 AM
Ottomata added a comment.Feb 27 2018, 2:56 PM

All for jmxtrans, varnishkafka, and kafkatee. Might want to keep zookeeper as a submodule. Just because there aren't many contributions, doesn't mean people don't use it :)

elukey added a comment.Feb 27 2018, 3:01 PM
In T188377#4006202, @Ottomata wrote:

All for jmxtrans, varnishkafka, and kafkatee. Might want to keep zookeeper as a submodule. Just because there aren't many contributions, doesn't mean people don't use it :)

Sure, but at some point we'll need to strive a balance between maintainability and submodules usage :)

Ottomata added a comment.Feb 27 2018, 3:04 PM

stars: https://github.com/wikimedia/puppet-zookeeper/stargazers
watchers: https://github.com/wikimedia/puppet-zookeeper/watchers
forks: https://github.com/wikimedia/puppet-zookeeper/network

:)

elukey added a comment.Feb 28 2018, 9:05 AM
In T188377#4006258, @Ottomata wrote:

stars: https://github.com/wikimedia/puppet-zookeeper/stargazers
watchers: https://github.com/wikimedia/puppet-zookeeper/watchers
forks: https://github.com/wikimedia/puppet-zookeeper/network

Half of the watchers are Wikimedia-related people, and 4 total forks :)

In my opinion having everything in ops/puppet is better, but if you feel strongly about a couple of modules we can skip them. As stated multiple times I support 100% the idea of sharing with others our work in a more modular way, but at the same time we also need to think about maintainability and convention (we are basically the only ones doing modules in ops). Having a module in operations/puppet doesn't mean that it will not be re-usable/sharable to others :)

Anyhow, let's do kafkatee/varnishkafka/jmxtrans for the moment. Would it be ok?

MoritzMuehlenhoff subscribed.Feb 28 2018, 11:02 AM
Ottomata added a comment.Feb 28 2018, 3:26 PM

Anyhow, let's do kafkatee/varnishkafka/jmxtrans for the moment. Would it be ok?

Ya, let's do these.

Half of the watchers are Wikimedia-related people, and 4 total forks :)

Yeah, but these are only people that have actually clicked the buttons. It doesn't show clones or downloads. The fact that we've actually gotten contributrions for cdh and zookeeper make me inclined to keep them as submodules.

we are basically the only ones doing modules in ops

nginx? :p

Having a module in operations/puppet doesn't mean that it will not be re-usable/sharable to others :)

It does kinda. No one is going to find or go into a subdirectory and pull out code, especially not if that code links to/uses other ops/puppet modules. Keeping them as a separate repository kinda forces us to code them in such a way that they are independent from other ops/puppet modules.

elukey added a comment.EditedFeb 28 2018, 3:32 PM
In T188377#4010694, @Ottomata wrote:

Anyhow, let's do kafkatee/varnishkafka/jmxtrans for the moment. Would it be ok?

Ya, let's do these.

I'll take this as a win and leave the other ones as submodules, I know that I am not going to convince you :D

• demon awarded a token.Feb 28 2018, 3:36 PM
MoritzMuehlenhoff triaged this task as Medium priority.Mar 1 2018, 9:33 AM
MoritzMuehlenhoff awarded a token.
Milimetric moved this task from Incoming to Dashiki on the Analytics board.Mar 1 2018, 6:14 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Mar 23 2018, 3:54 PM
Milimetric moved this task from Dashiki to Incoming on the Analytics board.Apr 2 2018, 3:32 PM
• Nuria lowered the priority of this task from Medium to Low.Apr 5 2018, 5:32 PM
• Nuria moved this task from Incoming to Operational Excellence on the Analytics board.
gerritbot subscribed.Jun 5 2018, 1:02 PM

Change 437467 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move the varnishkafka submodule to operations/puppet

https://gerrit.wikimedia.org/r/437467

gerritbot added a project: Patch-For-Review.Jun 5 2018, 1:02 PM
elukey added a project: Analytics-Kanban.Jun 6 2018, 2:42 PM
elukey moved this task from Next Up to In Code Review on the Analytics-Kanban board.
elukey claimed this task.Jun 6 2018, 4:05 PM
elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.Jun 7 2018, 7:24 AM
gerritbot added a comment.Jun 7 2018, 10:53 AM

Change 437950 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move the kafkatee submodule to operations/puppet

https://gerrit.wikimedia.org/r/437950

gerritbot added a comment.Jun 7 2018, 10:53 AM

Change 437951 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move the jmxtrans submodule to operations/puppet

https://gerrit.wikimedia.org/r/437951

elukey added a comment.Jun 7 2018, 1:56 PM

Ready to merge the code reviews for varnishkafka, kafkatee and jmxtrans, but it seems from the past experience that these actions break the puppet masters (prod and labs ones). Last time that we merged the mariadb module in operations/puppet puppet-merge broke and it was necessary to perform manual actions.

Andrew subscribed.Jun 11 2018, 6:12 PM

I don't mind having to manually fix some puppetmasters, although it would be nice to do them all at once :) I'll try cherry-picking a few of these changes and see if I can get a sense of what's likely to break.

gerritbot added a comment.Jun 12 2018, 1:13 PM

Change 439901 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move the nginx submodule to operations/puppet

https://gerrit.wikimedia.org/r/439901

gerritbot added a comment.Jun 12 2018, 1:17 PM

Change 439901 abandoned by Elukey:
Move the nginx submodule to operations/puppet

https://gerrit.wikimedia.org/r/439901

gerritbot added a comment.Jun 13 2018, 9:48 AM

Change 440080 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move the nginx submodule to operations/puppet

https://gerrit.wikimedia.org/r/440080

elukey added a subtask: T197503: Archive operations/puppet/varnishkafka repository.Jun 25 2018, 6:41 AM
gerritbot added a comment.Jun 25 2018, 2:01 PM

Change 441870 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move away jmxtrans/kafkatee's modules from environments/production

https://gerrit.wikimedia.org/r/441870

gerritbot added a comment.Jun 25 2018, 2:08 PM

Change 441870 merged by Elukey:
[operations/puppet@production] Move away jmxtrans/kafkatee's modules from environments/production

https://gerrit.wikimedia.org/r/441870

gerritbot added a comment.Jun 25 2018, 2:54 PM

Change 441879 had a related patch set uploaded (by Elukey; owner: Elukey):
[integration/config@master] Remove puppet submodules merged into operations/puppet

https://gerrit.wikimedia.org/r/441879

elukey added a comment.Jun 25 2018, 5:29 PM

All modules imported into operations/puppet, the remaining thing to do is cleaning up (subtasks).

elukey set the point value for this task to 8.Jun 26 2018, 7:26 AM
elukey moved this task from In Code Review to Done on the Analytics-Kanban board.
elukey closed subtask T197503: Archive operations/puppet/varnishkafka repository as Resolved.Jun 26 2018, 8:04 AM
elukey closed subtask T198097: Archive operations/puppet/jmxtrans repository as Resolved.
gerritbot added a comment.Jun 26 2018, 8:07 AM

Change 441879 abandoned by Elukey:
Remove puppet submodules merged into operations/puppet

Reason:
Yep! done as part of T198170

https://gerrit.wikimedia.org/r/441879

elukey closed subtask T198098: Archive operations/puppet/kafkatee repository as Resolved.Jun 26 2018, 8:09 AM
elukey moved this task from In Progress to Done on the User-Elukey board.Jun 26 2018, 2:30 PM
• Vvjjkkii reopened subtask T198098: Archive operations/puppet/kafkatee repository as Open.Jul 1 2018, 1:02 AM
• Vvjjkkii reopened subtask T198097: Archive operations/puppet/jmxtrans repository as Open.
• Vvjjkkii reopened subtask T197503: Archive operations/puppet/varnishkafka repository as Open.
gerritbot added a comment.Jul 2 2018, 7:32 AM

Change 437951 abandoned by Elukey:
Move the jmxtrans submodule to operations/puppet

https://gerrit.wikimedia.org/r/437951

gerritbot added a comment.Jul 2 2018, 7:32 AM

Change 437950 abandoned by Elukey:
Move the kafkatee submodule to operations/puppet

https://gerrit.wikimedia.org/r/437950

gerritbot added a comment.Jul 2 2018, 7:32 AM

Change 437467 abandoned by Elukey:
Move the varnishkafka submodule to operations/puppet

https://gerrit.wikimedia.org/r/437467

gerritbot added a comment.Jul 2 2018, 7:41 AM

Change 440080 abandoned by Elukey:
Move the nginx submodule to operations/puppet

https://gerrit.wikimedia.org/r/440080

hashar closed subtask T198097: Archive operations/puppet/jmxtrans repository as Resolved.Jul 2 2018, 7:01 PM
hashar closed subtask T197503: Archive operations/puppet/varnishkafka repository as Resolved.
elukey closed subtask T198098: Archive operations/puppet/kafkatee repository as Resolved.Jul 4 2018, 7:38 AM
• Nuria closed this task as Resolved.Jul 12 2018, 5:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL