Page MenuHomePhabricator
Create Task
Maniphest T121561

Encrypt Kafka traffic, and restrict access via ACLs
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

For now we're securing the caches' kafka broker traffic with IPSec ( T92602 ), but longer-term we need to move this to TLS.

We'd like to encrypt all Kafka traffic, both inter broker and between clients and brokers.

rough idea

  • Make cassandra-ca-manager script generic, and also make it output other useful certificate formats (.pem?) needed by non Java clients.
  • Use ca-manager script to generate CA certs, broker keys and client keys needed to configure Kafka TLS.
  • Client keys will be generated per logical client, not per client instance. E.g. varnishkafka-webrequest instances will all share the same client keys, distributed by puppet.
  • The CA certs will be distributed to all hosts that might run Kafka clients (e.g. Hadoop nodes), to support unauthenticated use cases (still encrypted).
  • Logical client keys will be distributed via puppet only to hosts that will run those specific clients.

Note that this plan doesn't yet consider encryption of traffic between Kafka and Zookeeper. Should we?

Details

ProjectBranchLines +/-Subject
labs/privatemaster+375 -28Update secrets/certificates with deployment-prep certs for TLS Kafka
operations/puppetproduction+38 -5Parameterize varnishkafka certificate name for easier setup in Cloud VPS.
operations/puppetproduction+2 -2Disable ssl for kafka-jumbo
operations/puppetproduction+1 -0chgrp server.properties to 'kafka' so daemon can read it
operations/puppetproduction+129 -79Puppetize SSL for Kafka broker
Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack created this task.Dec 15 2015, 7:00 PM
BBlack raised the priority of this task from to Needs Triage.
BBlack updated the task description. (Show Details)
BBlack added projects: Traffic, Analytics.
BBlack added a subscriber: BBlack.
Restricted Application added a project: SRE. · View Herald TranscriptDec 15 2015, 7:00 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Ottomata claimed this task.Dec 15 2015, 7:01 PM
Ottomata set Security to None.
BBlack added a project: Analytics-Clusters.Dec 15 2015, 7:02 PM
Ottomata renamed this task from Upgrade kafka for native TLS and secure the kafka traffic with it to Enable Kafka native TLS in 0.9 and secure the kafka traffic with it.Dec 15 2015, 7:03 PM
Milimetric moved this task from Incoming to temporary on the Analytics board.Jan 12 2016, 7:28 PM
Milimetric moved this task from temporary to Incoming on the Analytics board.Jan 12 2016, 7:35 PM
Milimetric moved this task from Incoming to temporary on the Analytics board.Jan 12 2016, 7:38 PM
Milimetric moved this task from temporary to Incoming on the Analytics board.Jan 12 2016, 7:43 PM
Milimetric triaged this task as Medium priority.Mar 7 2016, 5:15 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.
Nuria closed subtask T121562: Upgrade analytics-eqiad Kafka cluster to Kafka 0.9 as Resolved.May 30 2016, 9:52 PM
Ottomata removed Ottomata as the assignee of this task.Aug 8 2016, 7:50 PM
Ottomata added a subscriber: Ottomata.
BBlack moved this task from Triage to Watching on the Traffic board.Oct 4 2016, 12:47 PM
Nuria moved this task from Event Platform to Wikistats on the Analytics board.Oct 31 2016, 3:54 PM
Nuria moved this task from Wikistats to Dashiki on the Analytics board.
Ottomata added a parent task: T152015: Provision new Kafka cluster(s) with security features.Nov 30 2016, 4:56 PM
Ottomata renamed this task from Enable Kafka native TLS in 0.9 and secure the kafka traffic with it to Enable Kafka TLS and secure the kafka traffic with it.May 17 2017, 12:57 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
Ottomata added a comment.May 17 2017, 1:48 PM

It looks like a lot of the hard work for this has been done for Cassandra over in T108953 and T111113. Documentation for this is at https://wikitech.wikimedia.org/wiki/Cassandra/Tools/cassandra-ca-manager and https://wikitech.wikimedia.org/wiki/Cassandra#Installing_and_generating_certificates.

We can use that tool (perhaps after giving it a more generic name :) ), to generate signed certs, truststore, and keystore for each broker, add them to puppet private, distribute them using puppet secret(), and then point Kafka at the truststore and keystore.

Ottomata merged a task: Restricted Task.May 18 2017, 10:55 AM
Ottomata added subscribers: faidon, Nuria, elukey and 3 others.
Ottomata renamed this task from Enable Kafka TLS and secure the kafka traffic with it to Encrypt Kafka traffic, and restrict access via ACLs.May 18 2017, 11:01 AM
Ottomata updated the task description. (Show Details)
elukey added a project: User-Elukey.May 19 2017, 10:46 AM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.May 19 2017, 10:52 AM
Ottomata created subtask T166832: Import Kafka messages into HDFS authenticating with TLS/SSL.Jun 1 2017, 8:14 PM
Ottomata added a comment.Jun 7 2017, 2:40 PM

We should do some work to understand how ACLs work and what ACLs for what topics we should set in production.

elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.Jun 23 2017, 10:23 AM
elukey moved this task from In Progress to Analytics Backlog on the User-Elukey board.
Ottomata moved this task from Backlog to Q1 2020/2021 on the Analytics-Clusters board.Jul 11 2017, 3:12 PM
mforns edited projects, added Analytics-Kanban; removed Analytics.Jul 31 2017, 3:41 PM
mforns set the point value for this task to 0.
mforns moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.
mforns added a subtask: T126494: Send Mediawiki Kafka logs to Kafka jumbo cluster with TLS encryption.Jul 31 2017, 4:10 PM
elukey added a comment.Aug 4 2017, 9:50 AM
In T121561#3323871, @Ottomata wrote:

We should do some work to understand how ACLs work and what ACLs for what topics we should set in production.

This was done in https://phabricator.wikimedia.org/T167304

elukey added a comment.Aug 4 2017, 9:53 AM

Note that this plan doesn't yet consider encryption of traffic between Kafka and Zookeeper. Should we?

We've set up strict firewall rules for our zookeeper clusters to allow only few unix groups to be able to communicate with Zookeeper. This has been considered enough for our purposes, but we have to keep in mind that Kafka ACLs are stored in Zookeeper so future access to those groups will need to be carefully reviewed :)

elukey added a comment.Aug 4 2017, 10:02 AM

@Ottomata should we keep this task open given that we already have https://phabricator.wikimedia.org/T166167 ?

Ottomata added a comment.Aug 4 2017, 1:26 PM

Let's keep it open and use this task to track actually enabling TLS / ACLs for different clients.

elukey moved this task from Analytics Backlog to Keep an eye on it on the User-Elukey board.Aug 9 2017, 10:47 AM
gerritbot added a subscriber: gerritbot.Nov 29 2017, 9:02 PM

Change 394144 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Puppetize SSL for Kafka broker

https://gerrit.wikimedia.org/r/394144

gerritbot added a project: Patch-For-Review.Nov 29 2017, 9:02 PM
gerritbot added a comment.Nov 29 2017, 9:53 PM

Change 394144 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Puppetize SSL for Kafka broker

https://gerrit.wikimedia.org/r/394144

gerritbot added a comment.Nov 30 2017, 6:52 PM

Change 394144 merged by Ottomata:
[operations/puppet@production] Puppetize SSL for Kafka broker

https://gerrit.wikimedia.org/r/394144

gerritbot added a comment.Nov 30 2017, 6:59 PM

Change 394368 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] chgrp server.properties to 'kafka' so daemon can read it

https://gerrit.wikimedia.org/r/394368

gerritbot added a comment.Nov 30 2017, 6:59 PM

Change 394368 merged by Ottomata:
[operations/puppet@production] chgrp server.properties to 'kafka' so daemon can read it

https://gerrit.wikimedia.org/r/394368

gerritbot added a comment.Nov 30 2017, 7:51 PM

Change 394383 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Disable ssl for kafka-jumbo

https://gerrit.wikimedia.org/r/394383

gerritbot added a comment.Nov 30 2017, 7:52 PM

Change 394383 merged by Ottomata:
[operations/puppet@production] Disable ssl for kafka-jumbo

https://gerrit.wikimedia.org/r/394383

gerritbot added a comment.Jan 17 2018, 3:59 PM

Change 404698 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Parameterize varnishkafka certificate name for easier setup in Cloud VPS.

https://gerrit.wikimedia.org/r/404698

gerritbot added a comment.Jan 17 2018, 4:05 PM

Change 404698 merged by Ottomata:
[operations/puppet@production] Parameterize varnishkafka certificate name for easier setup in Cloud VPS.

https://gerrit.wikimedia.org/r/404698

gerritbot added a comment.Jan 17 2018, 4:28 PM

Change 404706 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[labs/private@master] Update secrets/certificates with deployment-prep certs for TLS Kafka

https://gerrit.wikimedia.org/r/404706

gerritbot added a comment.Jan 17 2018, 4:29 PM

Change 404706 merged by Ottomata:
[labs/private@master] Update secrets/certificates with deployment-prep certs for TLS Kafka

https://gerrit.wikimedia.org/r/404706

Ottomata added a project: Analytics.Mar 1 2018, 6:47 PM
Ottomata removed a project: Analytics-Kanban.Mar 1 2018, 6:54 PM
Milimetric moved this task from Dashiki to Incoming on the Analytics board.Apr 2 2018, 3:33 PM
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Nuria moved this task from Incoming to Kafka Work on the Analytics board.Apr 5 2018, 5:02 PM
mforns closed this task as Resolved.Apr 16 2018, 4:14 PM
mforns claimed this task.
Ottomata closed subtask T126494: Send Mediawiki Kafka logs to Kafka jumbo cluster with TLS encryption as Declined.May 29 2019, 9:49 AM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL