Page MenuHomePhabricator
Create Task
Maniphest T196081

Enable TLS and authorization for cross DC MirrorMaker
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Now that T193778: SSL and inter broker encryption for Kafka main is done, we can enable TLS for MirrorMaker. Once that is done, we can re-enable cross DC mirroring for job queue and change prop topics.

Details

SubjectRepoBranchLines +/-
Enable SSL for main-codfw -> main-eqiad Kafka MirrorMakeroperations/puppetproduction+3 -0
Move kafka_mirror_maker ssl path to /etc/kafka/mirror/ssloperations/puppetproduction+1 -1
Enable TLS MirrorMaker consumer for main-eqiad -> main-codfw MirrorMakeroperations/puppetproduction+3 -0
SSL for Kafka MirrorMakeroperations/puppetproduction+187 -51
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.May 31 2018, 3:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2018, 3:36 PM
Ottomata triaged this task as Medium priority.May 31 2018, 4:31 PM
Ottomata set the point value for this task to 8.
• Nuria raised the priority of this task from Medium to High.May 31 2018, 4:31 PM
• Nuria lowered the priority of this task from High to Medium.
• Nuria moved this task from Incoming to Kafka Work on the Analytics board.
• Pchelolo moved this task from Backlog to watching on the Services board.Jun 4 2018, 10:05 AM
• Pchelolo edited projects, added Services (watching); removed Services.
Ottomata merged a task: T194764: TLS encryption for cross DC Kafka main MirrorMaker instances.Jun 12 2018, 7:24 PM
Ottomata added a subscriber: akosiaris.
Ottomata added a comment.EditedJun 12 2018, 9:13 PM

Assuming the certificate CN is kafka_mirror_maker, we will need to add the following ACLs to clusters to which we connect over SSL port 9093:

kafka acls --add    --allow-principal User:ANONYMOUS             --topic '*' --group '*' --producer --consumer
kafka acls --add    --allow-principal User:CN=kafka_mirror_maker --topic '*' --group '*' --producer --consumer
Ottomata added a project: Analytics-Kanban.Jun 12 2018, 9:13 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Jun 13 2018, 4:48 PM

Change 440162 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] SSL for Kafka MirrorMaker

https://gerrit.wikimedia.org/r/440162

gerritbot added a project: Patch-For-Review.Jun 13 2018, 4:48 PM
Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jun 14 2018, 3:49 PM
Stashbot subscribed.Jun 14 2018, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-14T17:08:16Z] <ottomata> applying ACLs to Kafka main-codfw and main-eqiad - T196081

gerritbot added a comment.Jun 14 2018, 5:32 PM

Change 440162 merged by Ottomata:
[operations/puppet@production] SSL for Kafka MirrorMaker

https://gerrit.wikimedia.org/r/440162

gerritbot added a comment.Jun 14 2018, 5:36 PM

Change 440378 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable TLS MirrorMaker consumer for main-eqiad -> main-codfw MirrorMaker

https://gerrit.wikimedia.org/r/440378

gerritbot added a comment.Jun 14 2018, 5:45 PM

Change 440378 merged by Ottomata:
[operations/puppet@production] Enable TLS MirrorMaker consumer for main-eqiad -> main-codfw MirrorMaker

https://gerrit.wikimedia.org/r/440378

gerritbot added a comment.Jun 14 2018, 6:00 PM

Change 440381 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move kafka_mirror_maker ssl path to /etc/kafka/mirror/ssl

https://gerrit.wikimedia.org/r/440381

gerritbot added a comment.Jun 14 2018, 6:02 PM

Change 440381 merged by Ottomata:
[operations/puppet@production] Move kafka_mirror_maker ssl path to /etc/kafka/mirror/ssl

https://gerrit.wikimedia.org/r/440381

gerritbot added a comment.Jun 14 2018, 6:08 PM

Change 440384 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable SSL for main-codfw -> main-eqiad Kafka MirrorMaker

https://gerrit.wikimedia.org/r/440384

gerritbot added a comment.Jun 14 2018, 6:08 PM

Change 440384 merged by Ottomata:
[operations/puppet@production] Enable SSL for main-codfw -> main-eqiad Kafka MirrorMaker

https://gerrit.wikimedia.org/r/440384

Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.Jun 14 2018, 6:13 PM
• Nuria closed this task as Resolved.Jun 25 2018, 11:17 PM
• Vvjjkkii renamed this task from Enable TLS and authorization for cross DC MirrorMaker to mwbaaaaaaa.Jul 1 2018, 1:07 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Ottomata as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed the point value for this task.
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
• Community_Tech_bot renamed this task from mwbaaaaaaa to Enable TLS and authorization for cross DC MirrorMaker.Jul 1 2018, 6:23 AM
• Community_Tech_bot closed this task as Resolved.
• Community_Tech_bot assigned this task to Ottomata.
• Community_Tech_bot set the point value for this task to 8.
• Community_Tech_bot updated the task description. (Show Details)
• Community_Tech_bot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
• Community_Tech_bot added subscribers: gerritbot, Aklapper.
CommunityTechBot lowered the priority of this task from High to Medium.Jul 3 2018, 3:24 AM
• Pchelolo closed subtask T197254: Re-enable cross DC mirroring of job and change-prop Kafka topics over TLS as Resolved.Jul 30 2018, 8:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits