Page MenuHomePhabricator
Create Task
Maniphest T192831

Use profile and prometheus for role::kafka::main::broker
Closed, ResolvedPublic
Actions

Description

I'd like to reduce the number of moving parts for the upcoming main Kafka cluster upgrade. It will be easier to manage this upgrade if we switch as many things as we can before the actual Kafka version upgrade. This task is about applying profile::kafka::broker to Kafka main, while keeping the same version of Java and Kafka running there. This will enable prometheus based monitoring, and deprecate the jmxtrans/graphite monitoring.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata triaged this task as Medium priority.Apr 23 2018, 6:48 PM
Ottomata created this task.
gerritbot added a comment.Apr 23 2018, 6:53 PM

Change 428405 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Prep for profile::kafka::broker for main Kafka

https://gerrit.wikimedia.org/r/428405

gerritbot added a project: Patch-For-Review.Apr 23 2018, 6:53 PM
gerritbot added a comment.Apr 23 2018, 7:06 PM

Change 428405 merged by Ottomata:
[operations/puppet@production] Prep for profile::kafka::broker for main Kafka

https://gerrit.wikimedia.org/r/428405

gerritbot added a comment.Apr 23 2018, 7:18 PM

Change 428410 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use realm conditional in role::kafka::main::broker so we can test

https://gerrit.wikimedia.org/r/428410

gerritbot added a comment.Apr 23 2018, 7:22 PM

Change 428410 merged by Ottomata:
[operations/puppet@production] Use realm conditional in role::kafka::main::broker so we can test

https://gerrit.wikimedia.org/r/428410

gerritbot added a comment.Apr 23 2018, 7:35 PM

Change 428415 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include proper profile::kafka::broker class in Cloud VPS

https://gerrit.wikimedia.org/r/428415

gerritbot added a comment.Apr 23 2018, 7:36 PM

Change 428415 merged by Ottomata:
[operations/puppet@production] Include proper profile::kafka::broker class in Cloud VPS

https://gerrit.wikimedia.org/r/428415

gerritbot added a comment.Apr 23 2018, 7:45 PM

Change 428419 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Only require java.security if declared (with Kafka ssl_enabled)

https://gerrit.wikimedia.org/r/428419

gerritbot added a comment.Apr 23 2018, 7:50 PM

Change 428419 merged by Ottomata:
[operations/puppet@production] Only require java.security if declared (with Kafka ssl_enabled)

https://gerrit.wikimedia.org/r/428419

gerritbot added a comment.Apr 24 2018, 2:16 PM

Change 428642 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] role::kafka::main - only include MirrorMaker in prod

https://gerrit.wikimedia.org/r/428642

gerritbot added a comment.Apr 24 2018, 2:17 PM

Change 428642 merged by Ottomata:
[operations/puppet@production] role::kafka::main - only include MirrorMaker in prod

https://gerrit.wikimedia.org/r/428642

gerritbot added a comment.Apr 24 2018, 2:30 PM

Change 428647 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set default 'cluster' for profile::kafka::broker::monitoring

https://gerrit.wikimedia.org/r/428647

gerritbot added a comment.Apr 24 2018, 2:33 PM

Change 428647 merged by Ottomata:
[operations/puppet@production] Set default 'cluster' for profile::kafka::broker::monitoring

https://gerrit.wikimedia.org/r/428647

gerritbot added a comment.Apr 24 2018, 2:36 PM

Change 428648 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set defaults for replica_maxlag broker monitoring thresholds

https://gerrit.wikimedia.org/r/428648

gerritbot added a comment.Apr 24 2018, 2:37 PM

Change 428648 merged by Ottomata:
[operations/puppet@production] Set defaults for replica_maxlag broker monitoring thresholds

https://gerrit.wikimedia.org/r/428648

gerritbot added a comment.Apr 24 2018, 2:48 PM

Change 428654 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unneeded defaults from role/kafka/main.yaml hiera

https://gerrit.wikimedia.org/r/428654

gerritbot added a comment.Apr 24 2018, 2:49 PM

Change 428654 merged by Ottomata:
[operations/puppet@production] Remove unneeded defaults from role/kafka/main.yaml hiera

https://gerrit.wikimedia.org/r/428654

gerritbot added a comment.Apr 24 2018, 2:52 PM

Change 428656 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use profile::kafka::broker for main-codfw

https://gerrit.wikimedia.org/r/428656

gerritbot added a comment.Apr 24 2018, 2:58 PM

Change 428658 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add kafka_cluster label to Prometheus Kafka metrics

https://gerrit.wikimedia.org/r/428658

gerritbot added a comment.Apr 24 2018, 2:59 PM

Change 428658 merged by Ottomata:
[operations/puppet@production] Add kafka_cluster label to Prometheus Kafka metrics

https://gerrit.wikimedia.org/r/428658

gerritbot added a comment.Apr 24 2018, 4:35 PM

Change 428681 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use kafka_cluster label instead of cluster for broker alerts

https://gerrit.wikimedia.org/r/428681

gerritbot added a comment.Apr 24 2018, 4:37 PM

Change 428681 merged by Ottomata:
[operations/puppet@production] Use kafka_cluster label instead of cluster for broker alerts

https://gerrit.wikimedia.org/r/428681

gerritbot added a comment.Apr 24 2018, 5:33 PM

Change 428703 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use different GC opts for Kafka in Java 7 vs Java 8

https://gerrit.wikimedia.org/r/428703

gerritbot added a comment.Apr 24 2018, 5:36 PM

Change 428703 merged by Ottomata:
[operations/puppet@production] Use different GC opts for Kafka in Java 7 vs Java 8

https://gerrit.wikimedia.org/r/428703

Stashbot added a comment.Apr 24 2018, 5:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-24T17:42:16Z] <ottomata> temp disabling puppet on kafka200* to apply profile::kafka::broker in main-codfw T192831

gerritbot added a comment.Apr 24 2018, 5:42 PM

Change 428656 merged by Ottomata:
[operations/puppet@production] Use profile::kafka::broker for main-codfw

https://gerrit.wikimedia.org/r/428656

gerritbot added a comment.Apr 24 2018, 6:50 PM

Change 428711 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use profile::kafka::broker in main-eqiad

https://gerrit.wikimedia.org/r/428711

Stashbot added a comment.Apr 24 2018, 6:54 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-24T18:54:11Z] <ottomata> temp disabling puppet and applying profile::kafka::broker on kafka100* T192831

gerritbot added a comment.Apr 24 2018, 6:54 PM

Change 428711 merged by Ottomata:
[operations/puppet@production] Use profile::kafka::broker in main-eqiad

https://gerrit.wikimedia.org/r/428711

gerritbot added a comment.Apr 24 2018, 7:09 PM

Change 428717 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unused role::kafka::main::broker

https://gerrit.wikimedia.org/r/428717

gerritbot added a comment.Apr 24 2018, 8:33 PM

Change 428826 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix kafka alert dashboard link

https://gerrit.wikimedia.org/r/428826

gerritbot added a comment.Apr 24 2018, 8:33 PM

Change 428717 merged by Ottomata:
[operations/puppet@production] Remove unused role::kafka::main::broker

https://gerrit.wikimedia.org/r/428717

gerritbot added a comment.Apr 24 2018, 8:35 PM

Change 428826 merged by Ottomata:
[operations/puppet@production] Fix kafka alert dashboard link

https://gerrit.wikimedia.org/r/428826

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Apr 24 2018, 8:40 PM
• fdans moved this task from Incoming to Kafka Work on the Analytics board.Apr 26 2018, 4:35 PM
• Nuria closed this task as Resolved.May 8 2018, 10:45 PM
• Vvjjkkii renamed this task from Use profile and prometheus for role::kafka::main::broker to weeaaaaaaa.Jul 1 2018, 1:14 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Ottomata as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed the point value for this task.
• Vvjjkkii removed subscribers: Aklapper, gerritbot.
Samwilson renamed this task from weeaaaaaaa to Use profile and prometheus for role::kafka::main::broker .Jul 1 2018, 4:59 AM
Samwilson closed this task as Resolved.
Samwilson assigned this task to Ottomata.
Samwilson lowered the priority of this task from High to Medium.
Samwilson removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Samwilson updated the task description. (Show Details)
Samwilson added subscribers: GerritBot, Aklapper.
Aklapper edited subscribers, added: gerritbot; removed: GerritBot.May 16 2023, 10:18 AM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMay 16 2023, 10:18 AM
Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 10:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits