Page MenuHomePhabricator

SSL and inter broker encryption for Kafka main
Closed, ResolvedPublic8 Story Points

Description

T167039 is about upgrading Kafka main clusters to 1.x. This ticket is about enabling SSL and inter broker encryption after the 1.x upgrade is done.

https://docs.confluent.io/current/kafka/incremental-security-upgrade.html

Prep work

  • Kafka upgraded to 1.x T167039
  • Addition of main Kafka broker TLS keys and certs and ssl_password in hiera.

production upgrade plan [WIP]

This upgrade requires 2 rolling restarts of each broker in a Kafka cluster.

  1. To enable SSL port communication
  2. To set security.inter.broker.protocol=SSL

For main-codfw:

  1. Merge https://gerrit.wikimedia.org/r/#/c/436171/ (main-codfw). For each broker, run puppet to enable SSL listener and restart each broker:
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe  --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election

# Now proceed with next broker...
  1. Merge https://gerrit.wikimedia.org/r/#/c/434362/ (main-codfw). For each broker, run puppet to set default inter.broker.protocol.version=SSL and restart each broker:
sudo puppet agent -t
sudo service kafka restart
# wait until broker is back up and in ISRs, initiate election:
watch "kafka topics --describe  --topic eqiad.mediawiki.revision-create | grep -E 'Isr:.*1001.*$'"
kafka preferred-replica-election

# Now proceed with next broker...

Done!

Repeat the above for main-eqiad.

Event Timeline

Ottomata created this task.May 3 2018, 6:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2018, 6:37 PM
Ottomata updated the task description. (Show Details)May 3 2018, 6:39 PM
mforns triaged this task as Normal priority.May 7 2018, 3:26 PM
mforns moved this task from Incoming to Kafka Work on the Analytics board.
Ottomata updated the task description. (Show Details)May 21 2018, 4:03 PM
Ottomata updated the task description. (Show Details)May 21 2018, 5:16 PM

Change 434358 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] no-op: Set inter_broker_ssl_enabled false

https://gerrit.wikimedia.org/r/434358

Change 434358 merged by Ottomata:
[operations/puppet@production] no-op: Set inter_broker_ssl_enabled false

https://gerrit.wikimedia.org/r/434358

Change 434361 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable Kafka SSL listener for main-codfw

https://gerrit.wikimedia.org/r/434361

Change 434362 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable SSL inter.broker communication for Kafka main-codfw

https://gerrit.wikimedia.org/r/434362

Ottomata updated the task description. (Show Details)May 21 2018, 6:01 PM
Ottomata set the point value for this task to 8.
Ottomata updated the task description. (Show Details)
Ottomata added a project: Analytics-Kanban.

@elukey Done in deployment-prep, looks like it does for jumbo, works fine.

If you are available, what say you about doing this tomorrow or Wednesday in codfw. (Ping @Pchelolo @mobrovac.)

@elukey Done in deployment-prep, looks like it does for jumbo, works fine.
If you are available, what say you about doing this tomorrow or Wednesday in codfw. (Ping @Pchelolo @mobrovac.)

I am ok but the Services team is doing the offsite and probably they will not be available before next week :(

Ok, we will wait until next week. Let's try for Monday!

Change 434361 merged by Ottomata:
[operations/puppet@production] Enable Kafka SSL listener for main-codfw

https://gerrit.wikimedia.org/r/434361

Change 436097 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka - set super.users even if auth acls is not enabled

https://gerrit.wikimedia.org/r/436097

Change 436097 merged by Ottomata:
[operations/puppet@production] Kafka - set super.users even if auth acls is not enabled

https://gerrit.wikimedia.org/r/436097

Change 436103 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka vary ssl_client_auth only if auth_acls_enabled is true

https://gerrit.wikimedia.org/r/436103

Change 436103 merged by Ottomata:
[operations/puppet@production] Kafka vary ssl_client_auth only if auth_acls_enabled is true

https://gerrit.wikimedia.org/r/436103

Ottomata updated the task description. (Show Details)May 29 2018, 8:39 PM

Change 436165 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka - Don't manage Cluster ACLs

https://gerrit.wikimedia.org/r/436165

Change 436165 merged by Ottomata:
[operations/puppet@production] Kafka - Don't manage Cluster ACLs

https://gerrit.wikimedia.org/r/436165

Ottomata updated the task description. (Show Details)May 29 2018, 8:54 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-30T14:11:19Z] <ottomata> enabling SSL port for Kafka main-codfw cluster (take 2 :) ) T193778

Change 436294 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Don't enable auth_acls until all brokers have SSL ports open

https://gerrit.wikimedia.org/r/436294

Change 436294 merged by Ottomata:
[operations/puppet@production] Don't enable auth_acls until all brokers have SSL ports open

https://gerrit.wikimedia.org/r/436294

Change 434362 merged by Ottomata:
[operations/puppet@production] Enable SSL inter.broker communication for Kafka main-codfw

https://gerrit.wikimedia.org/r/434362

Change 436540 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable SSL port for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436540

Change 436541 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Enable inter broker SSL and auth acls for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436541

Change 436540 merged by Ottomata:
[operations/puppet@production] Enable SSL port for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436540

Mentioned in SAL (#wikimedia-operations) [2018-05-31T14:08:24Z] <ottomata> beginning restarts of Kafka main-eqiad to enable SSL port - T193778

Change 436541 merged by Ottomata:
[operations/puppet@production] Enable inter broker SSL and auth acls for Kafka main-eqiad

https://gerrit.wikimedia.org/r/436541

Nuria closed this task as Resolved.Jun 11 2018, 11:03 PM
Vvjjkkii renamed this task from SSL and inter broker encryption for Kafka main to lodaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Ottomata as the assignee of this task.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed the point value for this task.
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from lodaaaaaaa to SSL and inter broker encryption for Kafka main.Jul 2 2018, 3:12 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Ottomata.
CommunityTechBot lowered the priority of this task from High to Normal.
CommunityTechBot set the point value for this task to 8.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: gerritbot, Aklapper.