Page MenuHomePhabricator
Create Task
Maniphest T253058

DRY kafka broker declaration in helmfiles
Open, MediumPublic
Actions

Description

Puppet has the authoritative list of Kafka brokers. Helmfiles that use Kafka hardcode that list in, and when SRE changes kafka brokers (like in T279342), helmfiles must be updated too. This is error prone and can lead to problems if SRE is not aware of what services depend on Kafka.

We attempted to use puppet to render kafka broker info into the general.yaml value file, but this doesn't quite work, because the Helm charts would have to know how to use this, and the values vary per DC.

We should see if there is a DNS or LVS based solution for this, so we don't have to hardcode lists of kafka brokers.

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+50 -8eventgate: add kafka egress policy stanza
operations/deployment-chartsmaster+128 -0networkpolicy: add autogenerated egress rules
operations/puppetproduction+17 -4kubernetes::deployment_server: also add kafka broker, pass CIDRs
operations/puppetproduction+32 -1Render kafka cluster connection info in helmfile-defaults/general-*.yaml
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.May 18 2020, 7:20 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 18 2020, 7:20 PM
Milimetric triaged this task as Low priority.May 28 2020, 3:30 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
Ottomata mentioned this in T213561: Discovery for Kafka cluster brokers.Jan 14 2021, 7:57 PM
gerritbot added a comment.Jan 14 2021, 8:27 PM

Change 656253 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Render kafka cluster connection info in helmfile-defaults/general-*.yaml

https://gerrit.wikimedia.org/r/656253

gerritbot added a project: Patch-For-Review.Jan 14 2021, 8:27 PM
hnowlan added a subscriber: hnowlan.Jan 15 2021, 11:45 AM
gerritbot added a comment.Jan 20 2021, 3:12 PM

Change 656253 merged by Ottomata:
[operations/puppet@production] Render kafka cluster connection info in helmfile-defaults/general-*.yaml

https://gerrit.wikimedia.org/r/656253

hashar mentioned this in T272673: Puppet failling on deploy-1002.devtools.eqiad.wmflabs.Jan 22 2021, 9:07 AM
Ottomata renamed this task from DRY kafka broker declaration into helmfiles from puppet to DRY kafka broker declaration in helmfiles.Apr 16 2021, 3:55 PM
Ottomata raised the priority of this task from Low to Medium.
Ottomata updated the task description. (Show Details)
Ottomata added projects: SRE, serviceops.
Ottomata added subscribers: herron, fgiunchedi, colewhite.
Ottomata added subscribers: JMeybohm, akosiaris.Apr 16 2021, 4:36 PM

Actually, I'm not sure even just doing LVS would help here. The helmfiles networkpolicy explicitly lists IP addresses that the service can talk to. The broker IPs would still have to manually updated in networkpolicy in values.yaml file.

@akosiaris @JMeybohm any ideas?

akosiaris added a comment.Apr 19 2021, 4:07 PM

Hi!

Adopting the new functionality in networkpolicy resources has indeed created some tech debt. It's a tech debt we created on purpose while devoting resources to finalize the migration away from the old way of maintaining those networkpolicies. Now that that's gone, I want to revisit it and deduplicate it as much as possible.

I have a couple of approaches for that in mind, I 'll try and upload a couple of changes this week.

Ottomata added a comment.Apr 19 2021, 4:08 PM

<3

akosiaris claimed this task.Apr 19 2021, 4:22 PM
gerritbot added a comment.Apr 28 2021, 2:23 PM

Change 682971 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] kubernetes::deployment_server: also add kafka broker, pass CIDRs

https://gerrit.wikimedia.org/r/682971

gerritbot added a comment.Apr 28 2021, 4:55 PM

Change 683379 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] networkpolicy: add autogenerated egress rules

https://gerrit.wikimedia.org/r/683379

gerritbot added a comment.Apr 29 2021, 8:05 AM

Change 682971 merged by Giuseppe Lavagetto:

[operations/puppet@production] kubernetes::deployment_server: also add kafka broker, pass CIDRs

https://gerrit.wikimedia.org/r/682971

gerritbot added a comment.Apr 29 2021, 1:34 PM

Change 683379 merged by jenkins-bot:

[operations/deployment-charts@master] networkpolicy: add autogenerated egress rules

https://gerrit.wikimedia.org/r/683379

gerritbot added a comment.May 4 2021, 10:48 AM

Change 684855 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] eventgate: add kafka egress policy stanza

https://gerrit.wikimedia.org/r/684855

JMeybohm merged a task: T264076: envoy service proxy: Add networkpolicy egress rule for enabled listeners.Aug 30 2021, 9:23 AM
JMeybohm mentioned this in T264076: envoy service proxy: Add networkpolicy egress rule for enabled listeners.Aug 30 2021, 9:28 AM
BTullis added a subscriber: BTullis.Mar 24 2022, 5:10 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMar 24 2022, 5:10 PM
EChetty moved this task from Incoming to Ingest on the Data-Engineering board.Apr 24 2022, 6:52 AM
JArguello-WMF removed a project: Analytics.Jul 6 2022, 9:49 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:23 PM
jijiki moved this task from 🙈🙉🙊Backlog to 🥋Good First Task on the serviceops board.Nov 8 2022, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL