Page MenuHomePhabricator
Create Task
Maniphest T193778

SSL and inter broker encryption for Kafka main
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

T167039 is about upgrading Kafka main clusters to 1.x. This ticket is about enabling SSL and inter broker encryption after the 1.x upgrade is done.

https://docs.confluent.io/current/kafka/incremental-security-upgrade.html

Prep work

  • Kafka upgraded to 1.x T167039
  • Addition of main Kafka broker TLS keys and certs and ssl_password in hiera.

production upgrade plan [WIP]

This upgrade requires 2 rolling restarts of each broker in a Kafka cluster.

  1. To enable SSL port communication
  2. To set security.inter.broker.protocol=SSL

For main-codfw:

  1. Merge https://gerrit.wikimedia.org/r/#/c/436171/ (main-codfw). For each broker, run puppet to enable SSL listener and restart each broker:
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe  --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election

# Now proceed with next broker...
  1. Merge https://gerrit.wikimedia.org/r/#/c/434362/ (main-codfw). For each broker, run puppet to set default inter.broker.protocol.version=SSL and restart each broker:
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe  --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election

# Now proceed with next broker...

Done!

Repeat the above for main-eqiad.

Details

SubjectRepoBranchLines +/-
Enable inter broker SSL and auth acls for Kafka main-eqiadoperations/puppetproduction+0 -11
Enable SSL port for Kafka main-eqiadoperations/puppetproduction+1 -5
Enable SSL inter.broker communication for Kafka main-codfwoperations/puppetproduction+0 -7
Don't enable auth_acls until all brokers have SSL ports openoperations/puppetproduction+1 -0
Kafka - Don't manage Cluster ACLsoperations/puppetproduction+8 -25
Kafka vary ssl_client_auth only if auth_acls_enabled is trueoperations/puppetproduction+15 -6
Kafka - set super.users even if auth acls is not enabledoperations/puppetproduction+27 -32
Enable Kafka SSL listener for main-codfwoperations/puppetproduction+0 -9
no-op: Set inter_broker_ssl_enabled falseoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.May 3 2018, 6:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2018, 6:37 PM
Ottomata updated the task description. (Show Details)May 3 2018, 6:39 PM
Ottomata mentioned this in T167039: Upgrade Kafka on main cluster with security features.May 3 2018, 6:49 PM
Ottomata added a parent task: T167039: Upgrade Kafka on main cluster with security features.
mforns triaged this task as Medium priority.May 7 2018, 3:26 PM
mforns moved this task from Incoming to Kafka Work on the Analytics board.
Ottomata updated the task description. (Show Details)May 21 2018, 4:03 PM
Ottomata updated the task description. (Show Details)May 21 2018, 5:16 PM
gerritbot subscribed.May 21 2018, 5:41 PM

Change 434358 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] no-op: Set inter_broker_ssl_enabled false

https://gerrit.wikimedia.org/r/434358

gerritbot added a project: Patch-For-Review.May 21 2018, 5:41 PM
gerritbot added a comment.May 21 2018, 5:43 PM

Change 434358 merged by Ottomata:
[operations/puppet@production] no-op: Set inter_broker_ssl_enabled false

https://gerrit.wikimedia.org/r/434358

gerritbot added a comment.May 21 2018, 6:00 PM

Change 434361 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable Kafka SSL listener for main-codfw

https://gerrit.wikimedia.org/r/434361

gerritbot added a comment.May 21 2018, 6:00 PM

Change 434362 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable SSL inter.broker communication for Kafka main-codfw

https://gerrit.wikimedia.org/r/434362

Ottomata updated the task description. (Show Details)May 21 2018, 6:01 PM
Ottomata set the point value for this task to 8.
Ottomata updated the task description. (Show Details)
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.May 21 2018, 6:03 PM
Ottomata added subscribers: • Pchelolo, • mobrovac.May 21 2018, 7:39 PM

@elukey Done in deployment-prep, looks like it does for jumbo, works fine.

If you are available, what say you about doing this tomorrow or Wednesday in codfw. (Ping @Pchelolo @mobrovac.)

elukey added a comment.May 22 2018, 7:32 AM
In T193778#4221240, @Ottomata wrote:

@elukey Done in deployment-prep, looks like it does for jumbo, works fine.

If you are available, what say you about doing this tomorrow or Wednesday in codfw. (Ping @Pchelolo @mobrovac.)

I am ok but the Services team is doing the offsite and probably they will not be available before next week :(

Ottomata moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.May 22 2018, 3:05 PM
Ottomata added a comment.May 22 2018, 4:18 PM

Ok, we will wait until next week. Let's try for Monday!

gerritbot added a comment.May 29 2018, 5:59 PM

Change 434361 merged by Ottomata:
[operations/puppet@production] Enable Kafka SSL listener for main-codfw

https://gerrit.wikimedia.org/r/434361

gerritbot added a comment.May 29 2018, 7:57 PM

Change 436097 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka - set super.users even if auth acls is not enabled

https://gerrit.wikimedia.org/r/436097

gerritbot added a comment.May 29 2018, 8:02 PM

Change 436097 merged by Ottomata:
[operations/puppet@production] Kafka - set super.users even if auth acls is not enabled

https://gerrit.wikimedia.org/r/436097

gerritbot added a comment.May 29 2018, 8:08 PM

Change 436103 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka vary ssl_client_auth only if auth_acls_enabled is true

https://gerrit.wikimedia.org/r/436103

gerritbot added a comment.May 29 2018, 8:13 PM

Change 436103 merged by Ottomata:
[operations/puppet@production] Kafka vary ssl_client_auth only if auth_acls_enabled is true

https://gerrit.wikimedia.org/r/436103

Ottomata updated the task description. (Show Details)May 29 2018, 8:39 PM
gerritbot added a comment.May 29 2018, 8:40 PM

Change 436165 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka - Don't manage Cluster ACLs

https://gerrit.wikimedia.org/r/436165

gerritbot added a comment.May 29 2018, 8:44 PM

Change 436165 merged by Ottomata:
[operations/puppet@production] Kafka - Don't manage Cluster ACLs

https://gerrit.wikimedia.org/r/436165

Ottomata updated the task description. (Show Details)May 29 2018, 8:54 PM
Stashbot subscribed.May 30 2018, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-30T14:11:19Z] <ottomata> enabling SSL port for Kafka main-codfw cluster (take 2 :) ) T193778

gerritbot added a comment.May 30 2018, 2:16 PM

Change 436294 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Don't enable auth_acls until all brokers have SSL ports open

https://gerrit.wikimedia.org/r/436294

gerritbot added a comment.May 30 2018, 2:16 PM

Change 436294 merged by Ottomata:
[operations/puppet@production] Don't enable auth_acls until all brokers have SSL ports open

https://gerrit.wikimedia.org/r/436294

gerritbot added a comment.May 30 2018, 2:24 PM

Change 434362 merged by Ottomata:
[operations/puppet@production] Enable SSL inter.broker communication for Kafka main-codfw

https://gerrit.wikimedia.org/r/434362

gerritbot added a comment.May 31 2018, 2:02 PM

Change 436540 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable SSL port for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436540

gerritbot added a comment.May 31 2018, 2:02 PM

Change 436541 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable inter broker SSL and auth acls for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436541

gerritbot added a comment.May 31 2018, 2:04 PM

Change 436540 merged by Ottomata:
[operations/puppet@production] Enable SSL port for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436540

Stashbot added a comment.May 31 2018, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T14:08:24Z] <ottomata> beginning restarts of Kafka main-eqiad to enable SSL port - T193778

gerritbot added a comment.May 31 2018, 2:18 PM

Change 436541 merged by Ottomata:
[operations/puppet@production] Enable inter broker SSL and auth acls for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436541

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.May 31 2018, 3:35 PM
Ottomata mentioned this in T196081: Enable TLS and authorization for cross DC MirrorMaker.
• Nuria closed this task as Resolved.Jun 11 2018, 11:03 PM
• Vvjjkkii reopened subtask T194764: TLS encryption for cross DC Kafka main MirrorMaker instances as Open.Jul 1 2018, 1:09 AM
• Vvjjkkii renamed this task from SSL and inter broker encryption for Kafka main to lodaaaaaaa.Jul 1 2018, 1:12 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Ottomata as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed the point value for this task.
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
• Community_Tech_bot closed subtask T194764: TLS encryption for cross DC Kafka main MirrorMaker instances as Duplicate.Jul 1 2018, 6:23 AM
CommunityTechBot renamed this task from lodaaaaaaa to SSL and inter broker encryption for Kafka main.Jul 2 2018, 3:12 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Ottomata.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot set the point value for this task to 8.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added subscribers: gerritbot, Aklapper.
elukey mentioned this in T291905: Allow kafka clients to verify brokers hostnames when using SSL.Sep 28 2021, 9:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits