Page MenuHomePhabricator
Create Task
Maniphest T253058

DRY kafka broker declaration in helmfiles
Open, MediumPublic
Actions

Description

Puppet has the authoritative list of Kafka brokers. Helmfiles that use Kafka hardcode that list in, and when SRE changes kafka brokers (like in T279342), helmfiles must be updated too. This is error prone and can lead to problems if SRE is not aware of what services depend on Kafka.

Potential solution:

  • Kafka cluster and broker info is rendered into a default helmfile values file on deployment server by puppet (just like the service mesh definitions) (see kafka_config.rb for how puppet does this)
  • A modules/app/kafka_1.0.0.tpl that creates define(s) for kafka_bootstrap_servers for a specified kafka clusters and ports
  • Charts can then opt in to using the define in their app specific configuration templates

Some context in T213561: Discovery for Kafka cluster brokers as well

Details

SubjectRepoBranchLines +/-
eventgate-* - use kafka egress and service meshoperations/deployment-chartsmaster+88 -624
networkpolicy: add autogenerated egress rulesoperations/deployment-chartsmaster+128 -0
kubernetes::deployment_server: also add kafka broker, pass CIDRsoperations/puppetproduction+17 -4
Render kafka cluster connection info in helmfile-defaults/general-*.yamloperations/puppetproduction+32 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.May 18 2020, 7:20 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 18 2020, 7:20 PM
Milimetric triaged this task as Low priority.May 28 2020, 3:30 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
Ottomata mentioned this in T213561: Discovery for Kafka cluster brokers.Jan 14 2021, 7:57 PM
gerritbot added a comment.Jan 14 2021, 8:27 PM

Change 656253 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Render kafka cluster connection info in helmfile-defaults/general-*.yaml

https://gerrit.wikimedia.org/r/656253

gerritbot added a project: Patch-For-Review.Jan 14 2021, 8:27 PM
hnowlan subscribed.Jan 15 2021, 11:45 AM
gerritbot added a comment.Jan 20 2021, 3:12 PM

Change 656253 merged by Ottomata:
[operations/puppet@production] Render kafka cluster connection info in helmfile-defaults/general-*.yaml

https://gerrit.wikimedia.org/r/656253

hashar mentioned this in T272673: Puppet failling on deploy-1002.devtools.eqiad.wmflabs.Jan 22 2021, 9:07 AM
Ottomata renamed this task from DRY kafka broker declaration into helmfiles from puppet to DRY kafka broker declaration in helmfiles.Apr 16 2021, 3:55 PM
Ottomata raised the priority of this task from Low to Medium.
Ottomata updated the task description. (Show Details)
Ottomata added projects: SRE, serviceops.
Ottomata added subscribers: herron, fgiunchedi, colewhite.
Ottomata added subscribers: JMeybohm, akosiaris.Apr 16 2021, 4:36 PM

Actually, I'm not sure even just doing LVS would help here. The helmfiles networkpolicy explicitly lists IP addresses that the service can talk to. The broker IPs would still have to manually updated in networkpolicy in values.yaml file.

@akosiaris @JMeybohm any ideas?

akosiaris added a comment.Apr 19 2021, 4:07 PM

Hi!

Adopting the new functionality in networkpolicy resources has indeed created some tech debt. It's a tech debt we created on purpose while devoting resources to finalize the migration away from the old way of maintaining those networkpolicies. Now that that's gone, I want to revisit it and deduplicate it as much as possible.

I have a couple of approaches for that in mind, I 'll try and upload a couple of changes this week.

Ottomata added a comment.Apr 19 2021, 4:08 PM

<3

akosiaris claimed this task.Apr 19 2021, 4:22 PM
gerritbot added a comment.Apr 28 2021, 2:23 PM

Change 682971 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] kubernetes::deployment_server: also add kafka broker, pass CIDRs

https://gerrit.wikimedia.org/r/682971

gerritbot added a comment.Apr 28 2021, 4:55 PM

Change 683379 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] networkpolicy: add autogenerated egress rules

https://gerrit.wikimedia.org/r/683379

gerritbot added a comment.Apr 29 2021, 8:05 AM

Change 682971 merged by Giuseppe Lavagetto:

[operations/puppet@production] kubernetes::deployment_server: also add kafka broker, pass CIDRs

https://gerrit.wikimedia.org/r/682971

gerritbot added a comment.Apr 29 2021, 1:34 PM

Change 683379 merged by jenkins-bot:

[operations/deployment-charts@master] networkpolicy: add autogenerated egress rules

https://gerrit.wikimedia.org/r/683379

gerritbot added a comment.May 4 2021, 10:48 AM

Change 684855 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] eventgate: add kafka egress policy stanza

https://gerrit.wikimedia.org/r/684855

JMeybohm merged a task: T264076: envoy service proxy: Add networkpolicy egress rule for enabled listeners.Aug 30 2021, 9:23 AM
JMeybohm mentioned this in T264076: envoy service proxy: Add networkpolicy egress rule for enabled listeners.Aug 30 2021, 9:28 AM
BTullis subscribed.Mar 24 2022, 5:10 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMar 24 2022, 5:10 PM
• EChetty moved this task from Incoming (new tickets) to Apache Iceberg Migration on the Data-Engineering board.Apr 24 2022, 6:52 AM
JArguello-WMF removed a project: Analytics.Jul 6 2022, 9:49 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:23 PM
jijiki moved this task from 🙈🙉🙊Backlog to 🥋Good First Task on the serviceops board.Nov 8 2022, 5:11 PM
Ottomata mentioned this in T335024: Update eventgate and eventstreams helm chart to use automatic kafka egress networkpolicies and envoy service mesh.Apr 19 2023, 3:00 PM
gerritbot added a comment.Jun 14 2023, 9:05 PM

Change 930259 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-* - use kafka egress and service mesh

https://gerrit.wikimedia.org/r/930259

gerritbot added a comment.Jun 14 2023, 9:12 PM

Change 930259 abandoned by Ottomata:

[operations/deployment-charts@master] eventgate-* - use kafka egress and service mesh

Reason:

Using the older I885eec19dcfbb759036ea9976f49155a0923ec48 patchset chain

https://gerrit.wikimedia.org/r/930259

Ottomata added a comment.Jun 21 2023, 5:45 PM

Status update:

networkpolicy for Kafka brokers has been DRY, but referencing the hostnames for Kafka brokers for application config has not.

gmodena subscribed.Jun 27 2023, 3:14 PM
bking subscribed.Jun 27 2023, 4:41 PM
Ottomata updated the task description. (Show Details)Jun 27 2023, 4:49 PM
Ottomata updated the task description. (Show Details)
JArguello-WMF moved this task from Apache Iceberg Migration to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 11:33 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 4:30 PM
JArguello-WMF moved this task from Data Eng Backlog to Event Platform Backlog on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:38 PM
BTullis edited projects, added Data-Platform-SRE; removed Data-Engineering.Jul 14 2023, 11:46 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJul 14 2023, 11:46 PM
Aklapper removed a project: Patch-For-Review.Sep 17 2023, 4:04 PM
akosiaris removed akosiaris as the assignee of this task.Sep 18 2023, 5:23 PM
Ottomata updated the task description. (Show Details)Oct 27 2023, 1:43 PM
Ottomata mentioned this in T300102: Upgrade Kafka to from 1.x to later version.Oct 27 2023, 1:53 PM
lbowmaker removed a project: Data Engineering and Event Platform Team.Nov 10 2023, 2:29 PM
Gehel moved this task from Incoming to Toil / Automation on the Data-Platform-SRE board.Dec 6 2023, 1:55 PM
Ottomata mentioned this in T331894: Improve how we address outside k8s infrastructure from within charts (e.g. network policies).Dec 19 2023, 5:17 PM
BTullis added a comment.Feb 28 2024, 6:31 PM

I believe that this ticket will be invalidated by the approach that that has tested and agreed upon in T331894: Improve how we address outside k8s infrastructure from within charts (e.g. network policies).
Therefore, we might want to decline this ticket.

Ottomata added a comment.Feb 29 2024, 7:57 PM

+1, or add this as a subtask of that?

Either good with me!

Gehel added a parent task: T331894: Improve how we address outside k8s infrastructure from within charts (e.g. network policies).Mar 22 2024, 8:59 AM
brouberol subscribed.EditedMar 22 2024, 2:14 PM

Starting today (at least for the staging-codfw and dse-k8s-eqiad clusters), apps running in Kubernetes we can use Kubernetes service discovery to get the IPs of the Kafka clusters running outside of Kubernetes.

brouberol@deploy1002:~$ host kerberos-kdc.external-services 10.192.75.126
Using domain server:
Name: 10.192.75.126
Address: 10.192.75.126#53
Aliases:

Host kerberos-kdc.external-services not found: 3(NXDOMAIN)
brouberol@deploy1002:~$ host kerberos-kdc.external-services.svc.cluster.local 10.192.75.126
Using domain server:
Name: 10.192.75.126
Address: 10.192.75.126#53
Aliases:

kerberos-kdc.external-services.svc.cluster.local has address 10.192.48.190
kerberos-kdc.external-services.svc.cluster.local has address 10.64.0.112
kerberos-kdc.external-services.svc.cluster.local has IPv6 address 2620:0:861:101:10:64:0:112
kerberos-kdc.external-services.svc.cluster.local has IPv6 address 2620:0:860:104:10:192:48:190
brouberol@deploy1002:~$ host kafka-jumbo-eqiad.external-services.svc.cluster.local 10.192.75.126
Using domain server:
Name: 10.192.75.126
Address: 10.192.75.126#53
Aliases:

kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.136.11
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.134.9
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.32.106
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.135.16
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.130.10
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.48.140
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.131.16
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.48.121
kafka-jumbo-eqiad.external-services.svc.cluster.local has address 10.64.132.21
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:107:10:64:48:140
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:107:10:64:48:121
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:10e:10:64:135:16
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:10f:10:64:136:11
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:103:10:64:32:106
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:109:10:64:130:10
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:10d:10:64:134:9
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:10a:10:64:131:16
kafka-jumbo-eqiad.external-services.svc.cluster.local has IPv6 address 2620:0:861:10b:10:64:132:21

This means that we can then leverage the default behavior of client.dns.lookup librdkafka configuration (introduced in v2.2.0: https://github.com/confluentinc/librdkafka/commit/961946e55fb3f89eb782d4011af4bf5cd3c31f17) to have the client tray all IPs when the DNS resolve to multiple IPs:

~/wmf/puppet production *7 ❯ kafkacfg query --source librdkafka 'name=client.dns.lookup' -v 2.2.0 | jq '.[0]'
{
  "name": "client.dns.lookup",
  "override": null,
  "range": "use_all_dns_ips, resolve_canonical_bootstrap_servers_only",
  "default": "use_all_dns_ips",
  "importance": "low",
  "description": "Controls how the client uses DNS lookups. By default, when the lookup returns multiple IP addresses for a hostname, they will all be attempted for connection before the connection is considered failed. This applies to both bootstrap and advertised servers. If the value is set to `resolve_canonical_bootstrap_servers_only`, each entry will be resolved and expanded into a list of canonical names. NOTE: Default here is different from the Java client's default behavior, which connects only to the first IP address returned for a hostname.  <br>*Type: enum value*",
  "scope": "consumer"
}

The same behavior has been added to the Java clients shipped with kafka 2.6 https://docs.cloudera.com/runtime/7.2.15/kafka-configuring/topics/kafka-client-dns-lookup-property.html

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL