Page MenuHomePhabricator

Move Kafka Jumbo's TLS clients to the new bundle
Open, In Progress, MediumPublic

Description

The parent task describes the current migration of Kafka brokers to the new Kafka PKI Intermediate CA. We need to update Kafka TLS client configs to use a truststore/bundle that accepts TLS certificates signed by the new Intermediate or by the Puppet CA.

List of Jumbo clients:

  • FR kafkatee
  • SRE kafkatee
  • mirror maker
  • varishkafka
  • atskafka
  • gobblin
  • netflow
  • eventgate analytics

Quickly verified on kafka-jumbo1001 with netstat -tuap | grep :9093 | awk '{print $4" "$5}' | sort | uniq but please let me know if I am missing any.

Event Timeline

odimitrijevic edited projects, added Analytics-Radar; removed Analytics.
elukey changed the task status from Open to Stalled.Nov 24 2021, 3:59 PM

Setting this to stalled until we agree on https://phabricator.wikimedia.org/T296089

Change 742671 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] atskafka: use the same ca certificate as varnishkafka

https://gerrit.wikimedia.org/r/742671

Change 742671 merged by Elukey:

[operations/puppet@production] atskafka: use the same ca certificate as varnishkafka

https://gerrit.wikimedia.org/r/742671

Change 742747 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] varnishkafka: use new ca bundle instead of the Puppet one

https://gerrit.wikimedia.org/r/742747

elukey changed the task status from Stalled to In Progress.Nov 30 2021, 3:40 PM

Change 742753 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] netflow: move kafka config to new CA bundle

https://gerrit.wikimedia.org/r/742753

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

Hey @elukey, this should not be a problem however this is exactly the wrong time of year to mess with the kafkatee pipeline. Can we postpone until early January?

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

Hey @elukey, this should not be a problem however this is exactly the wrong time of year to mess with the kafkatee pipeline. Can we postpone until early January?

Sure makes sense, we can postpone it. I'll try to work on other clusters before Jumbo :)

elukey changed the task status from In Progress to Stalled.Nov 30 2021, 5:30 PM

Back to stalled, let's do it in January!

Change 742753 merged by Elukey:

[operations/puppet@production] netflow: move kafka config to new CA bundle

https://gerrit.wikimedia.org/r/742753

elukey changed the task status from Stalled to In Progress.Tue, Jan 11, 8:21 AM

Back to in-progress, the FR kafkatee instances moved to the new bundle!

Change 752992 had a related patch set uploaded (by Elukey; author: Elukey):

[eventgate-wikimedia@master] blubber: add wmf-certificates to the Docker images

https://gerrit.wikimedia.org/r/752992

Next steps:

Change 752992 merged by Ottomata:

[eventgate-wikimedia@master] blubber: add wmf-certificates to the Docker images

https://gerrit.wikimedia.org/r/752992

Change 753425 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: move eventgate-analytics* to the WMF CA cert bundle

https://gerrit.wikimedia.org/r/753425

Change 753428 had a related patch set uploaded (by Elukey; author: Elukey):

[mediawiki/services/eventstreams@master] blubber: deploy the wmf-certificates package in prod

https://gerrit.wikimedia.org/r/753428

Change 753428 merged by Elukey:

[mediawiki/services/eventstreams@master] blubber: deploy the wmf-certificates package in prod

https://gerrit.wikimedia.org/r/753428