Page MenuHomePhabricator
Create Task
Maniphest T336041

Bring kafka-jumbo10[09-15] into service
Closed, ResolvedPublic
Actions

Description

We need to bring these seven new kafka brokers into service.
They are currently in the insetup::data_engineering role.

These are to refresh kafka-jumbo100[1-6] which should be decommissioned afterwards.

  • kafka-jumbo1009
  • kafka-jumbo1010
  • kafka-jumbo1011
  • kafka-jumbo1012
  • kafka-jumbo1013
  • kafka-jumbo1014
  • kafka-jumbo1015

Steps:

  • Assign the role(kafka::jumbo::broker) role to the brokers
    • kafka-jumbo1010
    • kafka-jumbo1011
    • kafka-jumbo1012
    • kafka-jumbo1013
    • kafka-jumbo1014
    • kafka-jumbo1015
  • Provision kafka-jumbo1010
  • reach out to fundraising-tech-ops about configuring the new brokers https://phabricator.wikimedia.org/T346642
  • Restart producers and consumers to take kafka-jumbo1010 into their broker list
    • deployment-charts: eventgate-analytics-external
    • deployment-charts: eventgate-analytics
    • deployment-charts: eventstreams-internal
    • deployment-charts: mw-page-content-change-enrich
    • analytics/refinery : gobblin
    • puppet : O:cache::text : varnishkafka (taken care by puppet automatically)
  • Provision kafka-jumbo1011
  • Provision kafka-jumbo1012
  • Provision kafka-jumbo1013
  • Provision kafka-jumbo1014
  • Provision kafka-jumbo1015
  • Rolling restart the brokers to auth data fetching from all brokers
  • Restart producers and consumers to take kafka-jumbo1010->15 into their broker list
    • deployment-charts: eventgate-analytics-external
    • deployment-charts: eventgate-analytics
    • deployment-charts: eventstreams-internal
    • deployment-charts: mw-page-content-change-enrich
    • analytics/refinery : gobblin
    • puppet : O:cache::text : varnishkafka (taken care by puppet automatically)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.May 5 2023, 9:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2023, 9:55 AM
BTullis mentioned this in T336044: Decommission kafka-jumbo100[1-6].May 5 2023, 10:05 AM
BTullis added a project: Data-Platform-SRE.Jun 5 2023, 1:44 PM
JArguello-WMF removed a project: Shared-Data-Infrastructure.Jun 29 2023, 1:44 PM
JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 10:38 PM
JArguello-WMF removed a project: Data-Platform-SRE.Jun 29 2023, 10:56 PM
BTullis edited projects, added: Data-Platform-SRE; removed: Data-Engineering.Jul 14 2023, 11:55 PM
Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Aug 29 2023, 1:39 PM
brouberol claimed this task.Sep 14 2023, 3:16 PM
gerritbot added a comment.Sep 15 2023, 8:14 AM

Change 957861 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo101[0-5].eqiad.wmnet

https://gerrit.wikimedia.org/r/957861

gerritbot added a project: Patch-For-Review.Sep 15 2023, 8:14 AM
Gehel moved this task from Ready for Work to In Progress on the Data-Platform-SRE board.Sep 15 2023, 8:24 AM
brouberol added a subtask: T346425: Reassign partitions away from kafka-jumbo100[1-6] to kafka-jumbo10[07-15] brokers.Sep 15 2023, 8:28 AM
gerritbot added a comment.Sep 15 2023, 12:46 PM

Change 957861 abandoned by Brouberol:

[operations/puppet@production] Configure kafka-jumbo101[0-5].eqiad.wmnet

Reason:

Closing this one in favor or stacked changes: https://gerrit.wikimedia.org/r/c/operations/puppet/+/957918/

https://gerrit.wikimedia.org/r/957861

Maintenance_bot removed a project: Patch-For-Review.Sep 15 2023, 1:11 PM
gerritbot added a comment.Sep 15 2023, 1:15 PM

Change 957918 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1010.eqiad.wmnet

https://gerrit.wikimedia.org/r/957918

gerritbot added a project: Patch-For-Review.Sep 15 2023, 1:15 PM

Change 957919 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1011.eqiad.wmnet

https://gerrit.wikimedia.org/r/957919

gerritbot added a comment.Sep 15 2023, 1:15 PM

Change 957920 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1012.eqiad.wmnet

https://gerrit.wikimedia.org/r/957920

gerritbot added a comment.Sep 15 2023, 1:15 PM

Change 957921 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1013.eqiad.wmnet

https://gerrit.wikimedia.org/r/957921

gerritbot added a comment.Sep 15 2023, 1:15 PM

Change 957922 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1014.eqiad.wmnet

https://gerrit.wikimedia.org/r/957922

gerritbot added a comment.Sep 15 2023, 1:15 PM

Change 957923 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Configure kafka-jumbo1015.eqiad.wmnet

https://gerrit.wikimedia.org/r/957923

gerritbot added a comment.Sep 18 2023, 9:55 AM

Change 957918 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1010.eqiad.wmnet

https://gerrit.wikimedia.org/r/957918

gerritbot added a comment.Sep 18 2023, 10:16 AM

Change 958436 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Fix kafka-jumbo node regular expression

https://gerrit.wikimedia.org/r/958436

gerritbot added a comment.Sep 18 2023, 1:11 PM

Change 958436 merged by Brouberol:

[operations/puppet@production] Fix kafka-jumbo node regular expression

https://gerrit.wikimedia.org/r/958436

Stashbot added a comment.Sep 18 2023, 1:14 PM

Mentioned in SAL (#wikimedia-analytics) [2023-09-18T13:14:41Z] <brouberol> Puppet run successfully on kafka-jumbo1010.eqiad.wmnet. The kafka service is running. T336041

brouberol mentioned this in T346642: Add kafka-jumbo10[10-15].eqiad.wmnet in configuration.Sep 18 2023, 2:39 PM
brouberol updated the task description. (Show Details)Sep 18 2023, 2:44 PM
brouberol updated the task description. (Show Details)
gerritbot added a comment.Sep 18 2023, 2:57 PM

Change 958496 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] [eventgate-analytics-external] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958496

gerritbot added a comment.Sep 18 2023, 2:57 PM

Change 958498 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] [eventstream-internal] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958498

gerritbot added a comment.Sep 18 2023, 2:57 PM

Change 958499 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] [mw-page-content-change-enrich] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958499

brouberol updated the task description. (Show Details)Sep 18 2023, 3:16 PM
brouberol updated the task description. (Show Details)Sep 18 2023, 3:22 PM
gerritbot added a comment.Sep 18 2023, 3:30 PM

Change 958511 had a related patch set uploaded (by Brouberol; author: Brouberol):

[analytics/refinery@master] [Gobblin] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958511

gerritbot added a comment.Sep 18 2023, 3:32 PM

Change 958497 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] [eventgate-analytics] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958497

brouberol updated the task description. (Show Details)Sep 18 2023, 3:36 PM
brouberol updated the task description. (Show Details)
gerritbot added a comment.Sep 18 2023, 7:15 PM

Change 958496 merged by jenkins-bot:

[operations/deployment-charts@master] Add kafka-jumbo1010.eqiad.wmnet to apps config

https://gerrit.wikimedia.org/r/958496

Stashbot added a comment.Sep 19 2023, 7:59 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-19T07:59:24Z] <brouberol> redeploying eventgate-analytics in staging T336041

Stashbot added a comment.Sep 19 2023, 8:02 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-19T08:02:23Z] <brouberol> redeploying eventgate-analytics-external in staging T336041

Stashbot added a comment.Sep 19 2023, 8:05 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-19T08:05:14Z] <brouberol> redeploying eventstream-internal in staging T336041

Stashbot added a comment.Sep 19 2023, 8:17 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-19T08:17:29Z] <brouberol> redeploying eventstream-analytics in eqiad T336041

Stashbot added a comment.Sep 19 2023, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:18:42Z] <brouberol> redeploying eventstream-analytics in codfw T336041

Stashbot added a comment.Sep 19 2023, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:20:34Z] <brouberol> redeploying eventstream-analytics-external in eqiad T336041

Stashbot added a comment.Sep 19 2023, 8:21 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:21:41Z] <brouberol> redeploying eventstream-analytics-external in codfw T336041

Stashbot added a comment.Sep 19 2023, 8:23 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:23:10Z] <brouberol> redeploying eventstreams-internal in codfw T336041

Stashbot added a comment.Sep 19 2023, 8:24 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:24:12Z] <brouberol> redeploying eventstreams-internal in eqiad T336041

Stashbot added a comment.Sep 19 2023, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:25:38Z] <brouberol> redeploying mw-page-content-change-enrich in eqiad T336041

Stashbot added a comment.Sep 19 2023, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-19T08:26:36Z] <brouberol> redeploying mw-page-content-change-enrich in codfw T336041

brouberol updated the task description. (Show Details)Sep 19 2023, 8:27 AM
gerritbot added a comment.Sep 19 2023, 9:39 AM

Change 958511 merged by Brouberol:

[analytics/refinery@master] [Gobblin] Add kafka-jumbo1010 to config

https://gerrit.wikimedia.org/r/958511

brouberol moved this task from In Progress to Needs Review on the Data-Platform-SRE board.Sep 19 2023, 1:24 PM
gerritbot added a comment.Sep 19 2023, 1:29 PM

Change 957919 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1011.eqiad.wmnet

https://gerrit.wikimedia.org/r/957919

gerritbot added a comment.Sep 19 2023, 1:41 PM

Change 957920 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1012.eqiad.wmnet

https://gerrit.wikimedia.org/r/957920

gerritbot added a comment.Sep 19 2023, 1:44 PM

Change 957921 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1013.eqiad.wmnet

https://gerrit.wikimedia.org/r/957921

gerritbot added a comment.Sep 19 2023, 1:46 PM

Change 957922 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1014.eqiad.wmnet

https://gerrit.wikimedia.org/r/957922

gerritbot added a comment.Sep 19 2023, 1:50 PM

Change 957923 merged by Brouberol:

[operations/puppet@production] Configure kafka-jumbo1015.eqiad.wmnet

https://gerrit.wikimedia.org/r/957923

brouberol updated the task description. (Show Details)Sep 19 2023, 1:53 PM
brouberol updated the task description. (Show Details)
gerritbot added a comment.Sep 19 2023, 1:58 PM

Change 958938 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Add kafka-jumbo10[11-15].eqiad.wmnet to the apps broker list

https://gerrit.wikimedia.org/r/958938

gerritbot added a comment.Sep 19 2023, 2:00 PM

Change 958939 had a related patch set uploaded (by Brouberol; author: Brouberol):

[analytics/refinery@master] Add kafka-jumbo10[11-15].eqiad.wmnet to the gobblin broker list

https://gerrit.wikimedia.org/r/958939

Gehel moved this task from Needs Review to In Progress on the Data-Platform-SRE board.Sep 19 2023, 3:52 PM
gerritbot added a comment.Sep 19 2023, 7:33 PM

Change 958939 merged by Brouberol:

[analytics/refinery@master] Add kafka-jumbo10[11-15].eqiad.wmnet to the gobblin broker list

https://gerrit.wikimedia.org/r/958939

gerritbot added a comment.Sep 19 2023, 7:33 PM

Change 958938 merged by Brouberol:

[operations/deployment-charts@master] Add kafka-jumbo10[11-15].eqiad.wmnet to the apps broker list

https://gerrit.wikimedia.org/r/958938

brouberol updated the task description. (Show Details)Sep 20 2023, 10:00 AM
brouberol updated the task description. (Show Details)Sep 21 2023, 8:07 AM
Stashbot added a comment.Sep 21 2023, 8:10 AM

Mentioned in SAL (#wikimedia-analytics) [2023-09-21T08:10:08Z] <brouberol> redeploying eventgate-analytics in staging T336041

Stashbot added a comment.Sep 21 2023, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-21T08:12:18Z] <brouberol> redeploying eventgate-analytics-external in staging T336041

Stashbot added a comment.Sep 21 2023, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-21T08:13:22Z] <brouberol> redeploying eventstreams-internal in staging T336041

Stashbot added a comment.Sep 21 2023, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-21T08:14:10Z] <brouberol> redeploying mw-page-content-change-enrich in staging T336041

brouberol updated the task description. (Show Details)Sep 21 2023, 9:43 AM
brouberol closed this task as Resolved.Sep 21 2023, 9:45 AM
brouberol updated the task description. (Show Details)
gerritbot added a comment.Sep 21 2023, 11:48 AM

Change 958499 abandoned by Brouberol:

[operations/deployment-charts@master] [mw-page-content-change-enrich] Add kafka-jumbo1010 to config

Reason:

Superseeded by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/958496

https://gerrit.wikimedia.org/r/958499

gerritbot added a comment.Sep 21 2023, 11:48 AM

Change 958498 abandoned by Brouberol:

[operations/deployment-charts@master] [eventstream-internal] Add kafka-jumbo1010 to config

Reason:

Superseeded by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/958496

https://gerrit.wikimedia.org/r/958498

gerritbot added a comment.Sep 21 2023, 11:48 AM

Change 958497 abandoned by Brouberol:

[operations/deployment-charts@master] [eventgate-analytics] Add kafka-jumbo1010 to config

Reason:

Superseeded by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/958496

https://gerrit.wikimedia.org/r/958497

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2023, 12:11 PM
brouberol closed subtask T346425: Reassign partitions away from kafka-jumbo100[1-6] to kafka-jumbo10[07-15] brokers as Resolved.Oct 16 2023, 6:45 AM
brouberol moved this task from In Progress to Done on the Data-Platform-SRE board.Nov 6 2023, 1:36 PM
BTullis mentioned this in T352759: Decommission kafka-jumbo100[1-6].Dec 5 2023, 12:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits