Page MenuHomePhabricator
Create Task
Maniphest T252675

Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster
Closed, ResolvedPublic
Actions

Description

Set up Kafka on the new Jumbo Brokers.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Add AAAA/PTR records for kafka-jumbo100[7-9]Elukeyoperations/dnsmaster+6 -0
Replace kafka[12]00[123] with kafka-main* in analyitcs-in4/6 filtersElukeyoperations/homer/publicmaster+24 -24
Add kafka-jumbo100[7-9] to analytics-in4 and analytics-in6 filtersElukeyoperations/homer/publicmaster+12 -0
Add kafka-jumbo100[7-9] to network policy for eventgate-analytics and eventgate-analytics-externalOttomataoperations/deployment-chartsmaster+61 -19
Add kafka-jumbo100[89] into the jumbo-eqiad kafka clusterOttomataoperations/puppetproduction+11 -6
Use apt::package_from_component in confluent::kafka::commonOttomataoperations/puppetproduction+15 -45
apt: add thirdparty/confluent to buster-wikimediaOttomataoperations/puppetproduction+3 -11
Add kafka-jumbo1007 to jumbo-eqiad brokersOttomataoperations/puppetproduction+7 -2
Allow kafka brokers in the to talk to eacn other's prometheus jmx exporterOttomataoperations/puppetproduction+19 -9
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.May 13 2020, 1:50 PM
elukey added a comment.May 13 2020, 1:55 PM

One thing - 1007-9 have buster, so we'll need to adjust puppet to deploy openjdk-8 instead of 11. After this I think it should be a simple apply 1. and 2. :)

Second thing - please double check the hosts as well, I think the partitions are ok but for example /srv is not at its full size yet:

elukey@kafka-jumbo1007:~$ sudo lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root vg0 -wi-ao---- 357.07g
  srv  vg1 -wi-ao----  17.46t

elukey@kafka-jumbo1007:~$ sudo pvs
  PV         VG  Fmt  Attr PSize   PFree
  /dev/sda2  vg0 lvm2 a--  446.34g 89.27g
  /dev/sdb1  vg1 lvm2 a--  <21.83t <4.37t

Given how much space is stored in other brokers I don't think it will be an issue (to expand lvm+ext4 partition afterwards) but let me know your thoughts!

gerritbot added a comment.May 13 2020, 3:10 PM

Change 596232 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Allow kafka brokers in the to talk to eacn other's prometheus jmx exporter

https://gerrit.wikimedia.org/r/596232

gerritbot added a project: Patch-For-Review.May 13 2020, 3:10 PM
gerritbot added a comment.May 13 2020, 4:37 PM

Change 596232 merged by Ottomata:
[operations/puppet@production] Allow kafka brokers in the to talk to eacn other's prometheus jmx exporter

https://gerrit.wikimedia.org/r/596232

Ottomata added a comment.May 13 2020, 4:47 PM

Before doing any migration, I'm going to some delete old/temp/unused/empty topics:

-l
USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1
USER_REVISION_CREATE_OTTO1
WINDOWED_EDITS_OTTO1
__consumer_offsets
__transaction_state
apiaction
atskafka_test_webrequest_text
connect-test
does_not_exist
edisa.mediawiki.job.xxx
edisa.mediawiki.jobrefreshLinks
eqiad.swift.ottotest17.upload-complete
eventLogging-valid-mixed
eventLogging_valid_mixed
ksql__commands
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515085821960-changelog
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515085821960-repartition
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515085880366-changelog
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515085880366-repartition
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515086320017-changelog
ksql_query_CTAS_USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1-KSQL_Agg_Query_1515086320017-repartition
ksql_query_CTAS_WINDOWED_EDITS_OTTO1-KSQL_Agg_Query_1515087323836-changelog
ksql_query_CTAS_WINDOWED_EDITS_OTTO1-KSQL_Agg_Query_1515087323836-repartition
ksql_query_CTAS_WINDOWED_EDITS_OTTO1-KSQL_Agg_Query_1517342517779-changelog
ksql_query_CTAS_WINDOWED_EDITS_OTTO1-KSQL_Agg_Query_1517342517779-repartition
ksql_transient_191992346661370295_1517342539309-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_1938206695746484381_1515087269002-KSQL_Agg_Query_1515087269000-changelog
ksql_transient_1938206695746484381_1515087269002-KSQL_Agg_Query_1515087269000-repartition
ksql_transient_2553420146253236669_1515086157500-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_3413554931857454656_1515085910015-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_5179313399353440185_1515086454743-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_6131579099634200457_1517342471565-KSQL_Agg_Query_1517342471532-changelog
ksql_transient_6131579099634200457_1517342471565-KSQL_Agg_Query_1517342471532-repartition
ksql_transient_7684896129343784672_1517333884318-KSQL_Agg_Query_1517333884302-changelog
ksql_transient_7684896129343784672_1517333884318-KSQL_Agg_Query_1517333884302-repartition
ksql_transient_7774396194685631972_1515086325820-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_7953828792787172621_1517334071509-KSQL_Agg_Query_1517334071487-changelog
ksql_transient_7953828792787172621_1517334071509-KSQL_Agg_Query_1517334071487-repartition
ksql_transient_8218116125491884087_1515087342711-KSTREAM-REDUCE-STATE-STORE-0000000003-changelog
ksql_transient_8351551296540040971_1515087208641-KSQL_Agg_Query_1515087208636-changelog
ksql_transient_8351551296540040971_1515087208641-KSQL_Agg_Query_1515087208636-repartition
mediawiki.page-links-chan
mediawiki.revision-create
mediawiki_ApiAction
mediawiki_CirrusSearchRequestSet
otto4
otto5
otto_test5
revision-create
temp_NavigationTiming
temp_NavigationTiming_replay
test
test_otto0
test_otto1
test_otto2
test_otto3
test_otto4
test_otto5
virtualpageview
webrequest
wmf_netflow
Maintenance_bot removed a project: Patch-For-Review.May 13 2020, 5:10 PM
elukey mentioned this in T234629: Move the Analytics infrastructure to Debian Buster.May 14 2020, 6:56 AM
Ottomata added a project: Analytics-Kanban.May 14 2020, 2:35 PM
gerritbot added a comment.May 18 2020, 1:52 PM

Change 597061 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add kafka-jumbo1007 to jumbo-eqiad brokers

https://gerrit.wikimedia.org/r/597061

gerritbot added a project: Patch-For-Review.May 18 2020, 1:52 PM
gerritbot added a comment.May 18 2020, 1:57 PM

Change 597061 merged by Ottomata:
[operations/puppet@production] Add kafka-jumbo1007 to jumbo-eqiad brokers

https://gerrit.wikimedia.org/r/597061

Maintenance_bot removed a project: Patch-For-Review.May 18 2020, 2:10 PM
gerritbot added a comment.May 18 2020, 2:14 PM

Change 597076 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] apt: add thirdparty/confluent to buster-wikimedia

https://gerrit.wikimedia.org/r/597076

gerritbot added a project: Patch-For-Review.May 18 2020, 2:14 PM
gerritbot added a comment.May 18 2020, 2:45 PM

Change 597076 merged by Ottomata:
[operations/puppet@production] apt: add thirdparty/confluent to buster-wikimedia

https://gerrit.wikimedia.org/r/597076

gerritbot added a comment.May 18 2020, 3:01 PM

Change 597097 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use apt::package_from_component in confluent::kafka::common

https://gerrit.wikimedia.org/r/597097

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.May 18 2020, 4:06 PM
gerritbot added a comment.May 18 2020, 5:27 PM

Change 597097 merged by Ottomata:
[operations/puppet@production] Use apt::package_from_component in confluent::kafka::common

https://gerrit.wikimedia.org/r/597097

Maintenance_bot removed a project: Patch-For-Review.May 18 2020, 6:10 PM
gerritbot added a comment.May 18 2020, 7:15 PM

Change 597134 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add kafka-jumbo100[89] into the jumbo-eqiad kafka cluster

https://gerrit.wikimedia.org/r/597134

gerritbot added a project: Patch-For-Review.May 18 2020, 7:15 PM
gerritbot added a comment.May 18 2020, 7:18 PM

Change 597134 merged by Ottomata:
[operations/puppet@production] Add kafka-jumbo100[89] into the jumbo-eqiad kafka cluster

https://gerrit.wikimedia.org/r/597134

Maintenance_bot removed a project: Patch-For-Review.May 18 2020, 8:10 PM
Jgreen closed subtask T254257: adjust fundraising firewalls and kafkatee configuration to accomodate new kafka brokers kafka-jumbo100[789] as Resolved.Jun 2 2020, 8:37 PM
gerritbot added a comment.Jun 3 2020, 2:12 PM

Change 602087 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] Add kafka-jumbo100[7-9] to network policy for eventgate-analytics and eventgate-analytics-external

https://gerrit.wikimedia.org/r/602087

gerritbot added a project: Patch-For-Review.Jun 3 2020, 2:12 PM
gerritbot added a comment.Jun 5 2020, 11:23 AM

Change 602087 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add kafka-jumbo100[7-9] to network policy for eventgate-analytics and eventgate-analytics-external

https://gerrit.wikimedia.org/r/602087

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2020, 12:10 PM
elukey added a project: Analytics-Clusters.Jun 10 2020, 2:37 PM
elukey moved this task from Backlog to Q4 2019/2020 on the Analytics-Clusters board.
Ottomata added subscribers: • ayounsi, akosiaris.EditedJun 11 2020, 5:19 PM

@elukey @akosiaris @ayounsi I think the Analytics VLAN ACLs need to be adjusted to allow connections to these new hosts. This is currently causing some ingestion issues with newly created EventLogging topics. Would appreciate some help ASAP.

Specifically, Analytics VLAN should allow connections to kafka-jumbo100[1-9] on ports 9092 and 9093. I think 100[7-9] need to be added to this rule.

Thank you!

gerritbot added a comment.Jun 11 2020, 5:40 PM

Change 604810 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/homer/public@master] Add kafka-jumbo100[7-9] to analytics-in4 and analytics-in6 filters

https://gerrit.wikimedia.org/r/604810

gerritbot added a project: Patch-For-Review.Jun 11 2020, 5:40 PM
gerritbot added a comment.Jun 12 2020, 6:37 AM

Change 605098 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/homer/public@master] Replace kafka[12]00[123] with kafka-main* in analyitcs-in4/6 filters

https://gerrit.wikimedia.org/r/605098

gerritbot added a comment.Jun 12 2020, 8:43 AM

Change 604810 merged by Elukey:
[operations/homer/public@master] Add kafka-jumbo100[7-9] to analytics-in4 and analytics-in6 filters

https://gerrit.wikimedia.org/r/604810

gerritbot added a comment.Jun 12 2020, 8:43 AM

Change 605098 merged by Elukey:
[operations/homer/public@master] Replace kafka[12]00[123] with kafka-main* in analyitcs-in4/6 filters

https://gerrit.wikimedia.org/r/605098

Stashbot added a comment.Jun 12 2020, 8:48 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-12T08:48:37Z] <elukey> update cr1/cr2 analyitics filters for T252767 and T252675

Stashbot mentioned this in T252767: Move Archiva to Debian Buster.Jun 12 2020, 8:48 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 12 2020, 9:10 AM
gerritbot added a comment.Jun 12 2020, 1:50 PM

Change 605225 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add AAAA/PTR records for kafka-jumbo100[7-9]

https://gerrit.wikimedia.org/r/605225

gerritbot added a project: Patch-For-Review.Jun 12 2020, 1:51 PM
gerritbot added a comment.Jun 12 2020, 1:56 PM

Change 605225 merged by Elukey:
[operations/dns@master] Add AAAA/PTR records for kafka-jumbo100[7-9]

https://gerrit.wikimedia.org/r/605225

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2020, 2:10 PM
elukey added a comment.Jun 22 2020, 8:46 AM

I had a chat with Andrew, this task is now to be considered done. We'll move/shuffle partitions for optimal balance across brokers in another task.

elukey updated the task description. (Show Details)Jun 22 2020, 8:47 AM
elukey set Final Story Points to 8.
elukey moved this task from Next Up to Done on the Analytics-Kanban board.
elukey mentioned this in T255973: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers.Jun 22 2020, 8:49 AM
• Nuria added a comment.Jun 22 2020, 4:57 PM

And, to triple confirm, all new hosts have been added to analytics VLAN?

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
• Nuria closed this task as Resolved.Jul 6 2020, 10:12 PM
BTullis mentioned this in T346887: Monitor kafka topics with a replication factor of 1.Sep 20 2023, 1:47 PM
brouberol closed subtask T253058: DRY kafka broker declaration in helmfiles as Resolved.Feb 13 2026, 8:13 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits