Page MenuHomePhabricator
Create Task
Maniphest T291905

Allow kafka clients to verify brokers hostnames when using SSL
Closed, ResolvedPublic
Actions

Description

Kafka brokers have their certificates generated with a common CN per cluster, e.g. kafka_main-eqiad_broker.
This prevents the client to verify the hostname of the brokers it connects to:

openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -verify_hostname kafka-main2001.codfw.wmnet kafka-main2001.codfw.wmnet:9093 <<< "Q"
CONNECTED(00000003)
depth=0 CN = kafka_main-codfw_broker
verify error:num=62:Hostname mismatch
verify return:1
[...]

Regarding existing clients:

  • kafka-python requires to explicitly disable hostname verification, c.f. P17333
  • librdkafka has hostname verification disabled by default but can be enabled setting ssl.endpoint.identification.algorithm to HTTPS, c.f. P17334
  • java clients have not been tested

Ref:

Details

SubjectRepoBranchLines +/-
P:trafficserver::backend: use ca provided by P:base::certificatesoperations/puppetproduction+32 -1
profile::kafka::broker: add pki_intermediate_name parameteroperations/puppetproduction+2 -1
profile: Rsyslog omkafka configs use new ca bundleoperations/puppetproduction+9 -3
P:rsyslog::kafka_shipper: move Kafka TLS CA settings to the new bundleoperations/puppetproduction+3 -1
P:cache::kafka::Webrequest: always include profile::cache::kafka::certificateoperations/puppetproduction+1 -1
kubernetes: expose internal CA bundle to helmoperations/puppetproduction+4 -1
gobblin: use the new jks TLS bundle to validate certificatesanalytics/refinerymaster+2 -0
Move kafka-test brokers to the PKI intermediate CAoperations/puppetproduction+1 -1
Deploy the WMF Internal CAs bundle truststore to Hadoop workersoperations/puppetproduction+4 -0
Deploy the wmf_trusted_cas.jks bundle where Gobblin runsoperations/puppetproduction+4 -0
profile::cache::kafka::webrequest: add pki settingsoperations/puppetproduction+7 -1
Move coal, navtiming and statsv to the new CA bundleoperations/puppetproduction+3 -3
Update deployment-prep's profile::base::certificates settingsoperations/puppetproduction+0 -1
P:pki::client: manually deploy the root CA in cloudoperations/puppetproduction+24 -18
profile::base::certificates: vary trusted_certs on realmoperations/puppetproduction+20 -10
profile::cache::kafka::webrequest: move atskafka to new CA bundleoperations/puppetproduction+1 -1
kafkatee:instance: change TLS CA bundleoperations/puppetproduction+1 -1
Move kafka-test1006 to PKI-based broker certsoperations/puppetproduction+1 -1
Enable PKI-based TLS certificate for kafka-test1006operations/puppetproduction+1 -1
profile::base::certificates: add truststore passwordoperations/puppetproduction+9 -0
profile::kafka::mirror: add support for PKI-enabled truststoreoperations/puppetproduction+24 -14
sslcert::trusted_ca: add explicit ordering for jksoperations/puppetproduction+7 -6
sslcert::trusted_ca: fix file titleoperations/puppetproduction+1 -1
P:certificates: add defaults to cloudoperations/puppetproduction+3 -1
profile::kafka::broker: move to profile::base::certificates for pkioperations/puppetproduction+9 -10
profile::base::certificates: add sslcert::trusted_ca optionsoperations/puppetproduction+21 -4
profile::kafka::mirror: add settings to support the migration to PKIoperations/puppetproduction+20 -14
sslcert::trusted_ca: check if the bundle .pem is definedoperations/puppetproduction+8 -6
profile::kafka::broker: use a jks truststore for PKI TLS certsoperations/puppetproduction+5 -8
profile::kafka::broker: add conditions to truststore deploymentsoperations/puppetproduction+34 -25
Enable PKI TLS certificates for kafka-test1006operations/puppetproduction+2 -0
profile::kafka::broker: allow to override super_usersoperations/puppetproduction+18 -11
profile::kafka::broker: add truststore for pki-based tls certsoperations/puppetproduction+6 -4
Add sslcert::trusted_root_caoperations/puppetproduction+110 -0
role::kafka::test::broker: use PKI Kafka TLS certificatesoperations/puppetproduction+3 -0
profile::kafka::broker: add new Kafka PKI intermediate CA optionoperations/puppetproduction+61 -22
role::pki::multirootca: add kafka intermediate CA configoperations/puppetproduction+2 -0
profile::pki: add new kafka intermediate CA public certoperations/puppetproduction+22 -0
profile::pki::root_ca: add new kafka intermediate CAoperations/puppetproduction+1 -0
P:java: add Wikimedia_Internal_Root_CA to truststoreoperations/puppetproduction+9 -1
P:pki: deploy certifcates via wmf-certificatesoperations/puppetproduction+5 -11
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 28 2021, 1:36 PM

Change 735385 merged by Elukey:

[operations/puppet@production] role::pki::multirootca: add kafka intermediate CA config

https://gerrit.wikimedia.org/r/735385

gerritbot added a comment.Oct 29 2021, 7:13 AM

Change 735565 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: add new Kafka PKI intermediate CA option

https://gerrit.wikimedia.org/r/735565

gerritbot added a comment.Oct 29 2021, 7:21 AM

Change 735566 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::kafka::test::broker: use PKI Kafka TLS certificates

https://gerrit.wikimedia.org/r/735566

elukey added a comment.Oct 29 2021, 10:37 AM
In T291905#7435799, @jbond wrote:
In T291905#7435523, @jbond wrote:
In T291905#7431136, @elukey wrote:

To recap the next steps:

  • Add the cfssl CA cert to the base truststore of all jvms (this IIUC is already done, but John lemme know if it is not)

I thought it was but i just checked and it dosn't seem to be, will fix

this has now been added

Just realized that this may need a roll restart of all the Kafka clusters to allow their jvms to pick up the new truststore (IIRC it is not done automagically)

gerritbot added a comment.Oct 29 2021, 1:36 PM

Change 735565 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: add new Kafka PKI intermediate CA option

https://gerrit.wikimedia.org/r/735565

gerritbot added a comment.Oct 29 2021, 1:48 PM

Change 735566 merged by Elukey:

[operations/puppet@production] role::kafka::test::broker: use PKI Kafka TLS certificates

https://gerrit.wikimedia.org/r/735566

elukey added a comment.Oct 29 2021, 4:06 PM

A lot of progresses today with @jbond, here's a summary:

  • The new keystore contains the intermediate CA crt as we needed, tested with openssl s_client.
  • All the brokers of our clusters currently trust only the puppet CA, since they are configured with a truststore and this means that they don't look into the system cacert bundle.
  • We should create a new truststore with the Puppet CA and the Root PKI (not sure if we also need the intermediate in there), and distribute it to all brokers and clients.
  • After this, we should be able to flip one broker at the time without incurring in validation issues etc..
mforns moved this task from Incoming (new tickets) to Ops Week on the Data-Engineering board.Nov 1 2021, 5:00 PM
gerritbot added a comment.Nov 4 2021, 1:28 PM

Change 736765 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add sslcert::trusted_root_ca

https://gerrit.wikimedia.org/r/736765

gerritbot added a comment.Nov 4 2021, 2:51 PM

Change 736785 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: add truststore for pki-based tls certs

https://gerrit.wikimedia.org/r/736785

gerritbot added a comment.Nov 4 2021, 2:54 PM

Change 736786 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Enable PKI TLS certificates for kafka-test1006

https://gerrit.wikimedia.org/r/736786

gerritbot added a comment.Nov 4 2021, 3:38 PM

Change 736805 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: allow to override super_users

https://gerrit.wikimedia.org/r/736805

gerritbot added a comment.Nov 5 2021, 8:11 AM

Change 736765 merged by Elukey:

[operations/puppet@production] Add sslcert::trusted_root_ca

https://gerrit.wikimedia.org/r/736765

gerritbot added a comment.Nov 5 2021, 8:11 AM

Change 736785 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: add truststore for pki-based tls certs

https://gerrit.wikimedia.org/r/736785

gerritbot added a comment.Nov 5 2021, 8:13 AM

Change 736805 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: allow to override super_users

https://gerrit.wikimedia.org/r/736805

gerritbot added a comment.Nov 5 2021, 8:14 AM

Change 736786 merged by Elukey:

[operations/puppet@production] Enable PKI TLS certificates for kafka-test1006

https://gerrit.wikimedia.org/r/736786

gerritbot added a comment.Nov 5 2021, 9:01 AM

Change 736991 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: add conditions to truststore deployments

https://gerrit.wikimedia.org/r/736991

gerritbot added a comment.Nov 5 2021, 9:30 AM

Change 736991 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: add conditions to truststore deployments

https://gerrit.wikimedia.org/r/736991

gerritbot added a comment.Nov 5 2021, 3:24 PM

Change 737055 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: use a jks truststore for PKI TLS certs

https://gerrit.wikimedia.org/r/737055

gerritbot added a comment.Nov 5 2021, 3:36 PM

Change 737055 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: use a jks truststore for PKI TLS certs

https://gerrit.wikimedia.org/r/737055

gerritbot added a comment.Nov 5 2021, 4:39 PM

Change 737091 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::mirror: add settings to support the migration to PKI

https://gerrit.wikimedia.org/r/737091

gerritbot added a comment.Nov 5 2021, 5:03 PM

Change 737095 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] sslcert::trusted_ca: check if the bundle .pem is defined

https://gerrit.wikimedia.org/r/737095

elukey added a comment.Nov 5 2021, 5:23 PM

Finally some progress!

Some notes:

  • We have now a generic define to create .p12/.jks truststores containing the bundle Puppet-CA/Root-PKI in puppet
  • The pre-steps to do before updating a given kafka cluster should be:
    • Replace all occurrences of the truststore used by brokers with the one created by the aforementioned define, so that brokers will be able to accept TLS certificates signed from both CAs. This needs a cluster restart.
    • Replace all occurrences of the trustore for kafka mirror maker instances (if used)
    • Change the super.users list in all broker configs to allow every CN:hostname of the cluster (in addition to the current CN:clustername). This allows to swap TLS certs on brokers one at the time, and it requires a cluster restart.
gerritbot added a comment.Nov 8 2021, 2:36 PM

Change 737095 abandoned by Elukey:

[operations/puppet@production] sslcert::trusted_ca: check if the bundle .pem is defined

Reason:

https://gerrit.wikimedia.org/r/737095

gerritbot added a comment.Nov 8 2021, 2:36 PM

Change 737091 abandoned by Elukey:

[operations/puppet@production] profile::kafka::mirror: add settings to support the migration to PKI

Reason:

https://gerrit.wikimedia.org/r/737091

gerritbot added a comment.Nov 8 2021, 2:58 PM

Change 737403 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::base::certificates: add sslcert::trusted_ca options

https://gerrit.wikimedia.org/r/737403

gerritbot added a comment.Nov 8 2021, 3:26 PM

Change 737408 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:trafficserver::backend: use ca provided by P:base::certificates

https://gerrit.wikimedia.org/r/737408

gerritbot added a comment.Nov 8 2021, 6:40 PM

Change 737470 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: move to profile::base::certificates for pki

https://gerrit.wikimedia.org/r/737470

gerritbot added a comment.Nov 9 2021, 11:01 AM

Change 737403 merged by Elukey:

[operations/puppet@production] profile::base::certificates: add sslcert::trusted_ca options

https://gerrit.wikimedia.org/r/737403

gerritbot added a comment.Nov 9 2021, 11:08 AM

Change 737470 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: move to profile::base::certificates for pki

https://gerrit.wikimedia.org/r/737470

gerritbot added a comment.Nov 9 2021, 11:10 AM

Change 737644 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:certificates: add defaults to cloud

https://gerrit.wikimedia.org/r/737644

gerritbot added a comment.Nov 9 2021, 11:11 AM

Change 737644 merged by Jbond:

[operations/puppet@production] P:certificates: add defaults to cloud

https://gerrit.wikimedia.org/r/737644

gerritbot added a comment.Nov 9 2021, 11:12 AM

Change 737645 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] sslcert::trusted_ca: fix file title

https://gerrit.wikimedia.org/r/737645

gerritbot added a comment.Nov 9 2021, 11:18 AM

Change 737645 merged by Elukey:

[operations/puppet@production] sslcert::trusted_ca: fix file title

https://gerrit.wikimedia.org/r/737645

gerritbot added a comment.Nov 9 2021, 11:32 AM

Change 737652 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] sslcert::trusted_ca: add explicit ordering for jks

https://gerrit.wikimedia.org/r/737652

gerritbot added a comment.Nov 9 2021, 11:44 AM

Change 737652 merged by Elukey:

[operations/puppet@production] sslcert::trusted_ca: add explicit ordering for jks

https://gerrit.wikimedia.org/r/737652

gerritbot added a comment.Nov 9 2021, 12:19 PM

Change 737661 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::mirror: add support for PKI-enabled truststore

https://gerrit.wikimedia.org/r/737661

gerritbot added a comment.Nov 9 2021, 1:21 PM

Change 737661 merged by Elukey:

[operations/puppet@production] profile::kafka::mirror: add support for PKI-enabled truststore

https://gerrit.wikimedia.org/r/737661

gerritbot added a comment.Nov 9 2021, 1:29 PM

Change 737672 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::base::certificates: add truststore password

https://gerrit.wikimedia.org/r/737672

gerritbot added a comment.Nov 9 2021, 1:46 PM

Change 737672 merged by Elukey:

[operations/puppet@production] profile::base::certificates: add truststore password

https://gerrit.wikimedia.org/r/737672

gerritbot added a comment.Nov 9 2021, 2:23 PM

Change 737711 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Enable PKI-based TLS certificate for kafka-test1006

https://gerrit.wikimedia.org/r/737711

gerritbot added a comment.Nov 9 2021, 2:25 PM

Change 737711 merged by Elukey:

[operations/puppet@production] Enable PKI-based TLS certificate for kafka-test1006

https://gerrit.wikimedia.org/r/737711

gerritbot added a comment.Nov 10 2021, 2:12 PM

Change 737920 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move kafka-test1006 to PKI-based broker certs

https://gerrit.wikimedia.org/r/737920

gerritbot added a comment.Nov 10 2021, 2:14 PM

Change 737920 merged by Elukey:

[operations/puppet@production] Move kafka-test1006 to PKI-based broker certs

https://gerrit.wikimedia.org/r/737920

elukey added a comment.Nov 10 2021, 2:38 PM

Finally kafka-test1006 is running with a PKI kafka intermediate cert, and the rest of the cluster works fine as well (still on puppet-based certs). Mirror Maker seems working fine with the new truststore (accepting both Puppet CA and PKI Root CA certs).

The procedure that we thought seems working, but before starting to move clusters we'll need to work on clients, to move them to the new bundle/truststore.

gerritbot added a comment.Nov 10 2021, 2:47 PM

Change 737923 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kafkatee:instance: change TLS CA bundle

https://gerrit.wikimedia.org/r/737923

gerritbot added a comment.Nov 10 2021, 3:06 PM

Change 737931 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::kafka::webrequest: move atskafka to new CA bundle

https://gerrit.wikimedia.org/r/737931

gerritbot added a comment.Nov 10 2021, 4:22 PM

Change 737923 merged by Elukey:

[operations/puppet@production] kafkatee:instance: change TLS CA bundle

https://gerrit.wikimedia.org/r/737923

Stashbot added a comment.Nov 10 2021, 4:26 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-10T16:26:43Z] <elukey> move kafkatee instances (analytics-test,centralog) to the new CA bundle - T291905

gerritbot added a comment.Nov 10 2021, 4:27 PM

Change 737931 merged by Elukey:

[operations/puppet@production] profile::cache::kafka::webrequest: move atskafka to new CA bundle

https://gerrit.wikimedia.org/r/737931

Stashbot added a comment.Nov 10 2021, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-10T16:28:40Z] <elukey> move atskafka to the new CA bundle - T291905

gerritbot added a comment.Nov 10 2021, 4:52 PM

Change 737970 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move coal and navtiming to the new CA bundle

https://gerrit.wikimedia.org/r/737970

gerritbot added a comment.Nov 10 2021, 6:47 PM

Change 737983 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::base::certificates: vary trusted_certs on realm

https://gerrit.wikimedia.org/r/737983

elukey added a comment.Nov 11 2021, 7:37 AM

First thing to follow up - deployment-prep:

  1. we have a kafka cluster in there (for example, to test evengate, etc..). Should we move it to PKI (if available) or just keep using the puppet certs?
  2. we currently deploy under /etc/ssl/localcerts the bundle of the production's puppet ca and root pki prod ca, that of course don't work in cloud.
Ottomata added a comment.Nov 12 2021, 2:22 PM

https://wikitech.wikimedia.org/wiki/PKI/Cloud ? maybe we can make it work?

elukey added a comment.Nov 12 2021, 2:29 PM
In T291905#7500211, @Ottomata wrote:

https://wikitech.wikimedia.org/wiki/PKI/Cloud ? maybe we can make it work?

I had a chat with @jbond and it should be available for deployment-prep, but we need something like https://gerrit.wikimedia.org/r/737983 first (to make sure that clients can trust the right CAs etc..). The only thing that we may need to add to profile::kafka::broker is the choice of the intermediate CA name to use, but should be easy enough!

gerritbot added a comment.Nov 15 2021, 3:53 PM

Change 737983 merged by Elukey:

[operations/puppet@production] profile::base::certificates: vary trusted_certs on realm

https://gerrit.wikimedia.org/r/737983

gerritbot added a comment.Nov 15 2021, 4:28 PM

Change 738958 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Update deployment-prep's profile::base::certificates settings

https://gerrit.wikimedia.org/r/738958

gerritbot added a comment.Nov 15 2021, 4:31 PM

Change 738958 merged by Elukey:

[operations/puppet@production] Update deployment-prep's profile::base::certificates settings

https://gerrit.wikimedia.org/r/738958

gerritbot added a comment.Nov 16 2021, 2:52 PM

Change 739266 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:pki::client: manually deploy the root CA in cloud

https://gerrit.wikimedia.org/r/739266

gerritbot added a comment.Nov 16 2021, 3:00 PM

Change 739266 merged by Jbond:

[operations/puppet@production] P:pki::client: manually deploy the root CA in cloud

https://gerrit.wikimedia.org/r/739266

gerritbot added a comment.Nov 17 2021, 7:47 AM

Change 737970 merged by Elukey:

[operations/puppet@production] Move coal, navtiming and statsv to the new CA bundle

https://gerrit.wikimedia.org/r/737970

gerritbot added a comment.Nov 17 2021, 7:55 AM

Change 739463 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::rsyslog: move Kafka TLS CA settings to the new bundle

https://gerrit.wikimedia.org/r/739463

gerritbot added a comment.Nov 17 2021, 9:41 AM

Change 739475 had a related patch set uploaded (by Elukey; author: Elukey):

[analytics/refinery@master] gobblin: use the new jks TLS bundle to validate certificates

https://gerrit.wikimedia.org/r/739475

gerritbot added a comment.Nov 17 2021, 9:44 AM

Change 739476 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Deploy the wmf_trusted_cas.jks bundle where Gobblin runs

https://gerrit.wikimedia.org/r/739476

gerritbot added a comment.Nov 18 2021, 2:22 PM

Change 739806 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::kafka::webrequest: add pki settings

https://gerrit.wikimedia.org/r/739806

gerritbot added a comment.Nov 18 2021, 5:21 PM

Change 739806 merged by Elukey:

[operations/puppet@production] profile::cache::kafka::webrequest: add pki settings

https://gerrit.wikimedia.org/r/739806

gerritbot added a comment.Nov 19 2021, 7:34 AM

Change 740083 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kubernetes: expose internal CA bundle to helm

https://gerrit.wikimedia.org/r/740083

gerritbot added a comment.Nov 19 2021, 7:37 AM

Change 739476 merged by Elukey:

[operations/puppet@production] Deploy the wmf_trusted_cas.jks bundle where Gobblin runs

https://gerrit.wikimedia.org/r/739476

gerritbot added a comment.Nov 19 2021, 8:00 AM

Change 740086 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Deploy the WMF Internal CAs bundle truststore to Hadoop test workers

https://gerrit.wikimedia.org/r/740086

gerritbot added a comment.Nov 19 2021, 8:04 AM

Change 740086 merged by Elukey:

[operations/puppet@production] Deploy the WMF Internal CAs bundle truststore to Hadoop workers

https://gerrit.wikimedia.org/r/740086

elukey added a comment.Nov 19 2021, 8:18 AM

At this point we need to migrate all Kafka client using TLS to the new bundle before proceeding further with clusters. I think it is best to open a task for each cluster as sub-task.

gerritbot added a comment.Nov 19 2021, 8:36 AM

Change 740091 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move kafka-test brokers to the PKI intermediate CA

https://gerrit.wikimedia.org/r/740091

gerritbot added a comment.Nov 19 2021, 8:37 AM

Change 740091 merged by Elukey:

[operations/puppet@production] Move kafka-test brokers to the PKI intermediate CA

https://gerrit.wikimedia.org/r/740091

gerritbot added a comment.Nov 19 2021, 8:41 AM

Change 739475 merged by Joal:

[analytics/refinery@master] gobblin: use the new jks TLS bundle to validate certificates

https://gerrit.wikimedia.org/r/739475

elukey added a comment.Nov 19 2021, 9:15 AM
elukey@kafka-test1006:~$ openssl s_client -CAfile /etc/ssl/localcerts/wmf_trusted_root_CAs.pem -verify_hostname kafka-test1006.eqiad.wmnet kafka-test1006.eqiad.wmnet:9093 <<< "Q"
[..]

Certificate chain
 0 s:CN = kafka-test1006.eqiad.wmnet
   i:C = US, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = SRE Foundations, CN = kafka
 1 s:C = US, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = SRE Foundations, CN = kafka
   i:C = US, ST = California, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = Cloud Services, CN = Wikimedia_Internal_Root_CA

[..]
---
SSL handshake has read 2263 bytes and written 456 bytes
Verification: OK
Verified peername: kafka-test1006.eqiad.wmnet
---
[..]
DONE

Kafka test migration worked nicely :)

gerritbot added a comment.Nov 19 2021, 10:06 AM

Change 740083 abandoned by Elukey:

[operations/puppet@production] kubernetes: expose internal CA bundle to helm

Reason:

Not needed!

https://gerrit.wikimedia.org/r/740083

elukey added a comment.Nov 19 2021, 3:35 PM

Let's wait for T296089 before proceeding :)

elukey changed the status of subtask T296064: Move Kafka Jumbo's TLS clients to the new bundle from Open to Stalled.Nov 24 2021, 3:59 PM
gerritbot added a comment.Nov 29 2021, 3:47 PM

Change 742482 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:cache::kafka::Webrequest: always include profile::cache::kafka::certificate

https://gerrit.wikimedia.org/r/742482

gerritbot added a comment.Nov 29 2021, 3:49 PM

Change 742482 merged by Jbond:

[operations/puppet@production] P:cache::kafka::Webrequest: always include profile::cache::kafka::certificate

https://gerrit.wikimedia.org/r/742482

elukey changed the status of subtask T296064: Move Kafka Jumbo's TLS clients to the new bundle from Stalled to In Progress.Nov 30 2021, 3:40 PM
elukey changed the status of subtask T296064: Move Kafka Jumbo's TLS clients to the new bundle from In Progress to Stalled.Nov 30 2021, 5:30 PM
elukey changed the status of subtask T296064: Move Kafka Jumbo's TLS clients to the new bundle from Stalled to In Progress.Jan 11 2022, 8:21 AM
Restricted Application removed a project: Data-Engineering. · View Herald TranscriptJan 11 2022, 8:21 AM
Maintenance_bot added a project: Data-Engineering.Jan 11 2022, 8:45 AM
elukey closed subtask T299409: Allow kafka brokers to reload the TLS keystore as Resolved.Jan 24 2022, 12:13 PM
gerritbot added a comment.Jan 27 2022, 1:39 PM

Change 739463 merged by Elukey:

[operations/puppet@production] P:rsyslog::kafka_shipper: move Kafka TLS CA settings to the new bundle

https://gerrit.wikimedia.org/r/739463

gerritbot added a comment.Jan 28 2022, 7:12 AM

Change 757800 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka::broker: add pki_intermediate_name parameter

https://gerrit.wikimedia.org/r/757800

gerritbot added a comment.Jan 28 2022, 10:11 AM

Change 757800 abandoned by Elukey:

[operations/puppet@production] profile::kafka::broker: add pki_intermediate_name parameter

Reason:

Not needed since John is awesome

https://gerrit.wikimedia.org/r/757800

gerritbot added a comment.Mar 23 2022, 4:51 PM

Change 773285 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: Rsyslog omkafka configs use new ca bundle

https://gerrit.wikimedia.org/r/773285

gerritbot added a comment.Mar 24 2022, 3:18 PM

Change 773285 merged by Cwhite:

[operations/puppet@production] profile: Rsyslog omkafka configs use new ca bundle

https://gerrit.wikimedia.org/r/773285

elukey closed this task as Resolved.Oct 6 2022, 12:56 PM
elukey claimed this task.

The kafka logging clusters have the new PKI configuration:

openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -verify_hostname kafka-logging1001.eqiad.wmnet kafka-logging1001.eqiad.wmnet:9093
[..]
SSL handshake has read 2270 bytes and written 459 bytes
Verification: OK
Verified peername: kafka-logging1001.eqiad.wmnet
[..]

The issue is solved for a real production cluster, the remaining work is in the subtasks (for Kafka Jumbo and main). We can close this task and follow up in the other ones.

colewhite closed subtask T300130: Move Kafka logging to the new intermediate PKI as Resolved.Feb 7 2023, 3:52 PM
elukey closed subtask T319372: Move Kafka main to the new intermediate PKI CA as Resolved.Apr 13 2023, 3:51 PM
elukey closed subtask T296064: Move Kafka Jumbo's TLS clients to the new bundle as Resolved.Apr 18 2023, 11:01 AM
gerritbot added a comment.Nov 28 2023, 6:39 PM

Change 737408 abandoned by Jbond:

[operations/puppet@production] P:trafficserver::backend: use ca provided by P:base::certificates

Reason:

this has since been implmented

https://gerrit.wikimedia.org/r/737408

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits