Page MenuHomePhabricator
Create Task
Maniphest T296064

Move Kafka Jumbo's TLS clients to the new bundle
Open, In Progress, MediumPublic
Actions

Description

The parent task describes the current migration of Kafka brokers to the new Kafka PKI Intermediate CA. We need to update Kafka TLS client configs to use a truststore/bundle that accepts TLS certificates signed by the new Intermediate or by the Puppet CA.

List of Jumbo clients:

  • FR kafkatee
  • SRE kafkatee
  • mirror maker
  • varishkafka
  • atskafka
  • gobblin
  • netflow
  • eventgate analytics

Quickly verified on kafka-jumbo1001 with netstat -tuap | grep :9093 | awk '{print $4" "$5}' | sort | uniq but please let me know if I am missing any.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0role::kafka::jumbo::broker: enable PKI migration settings
operations/deployment-chartsmaster+4 -2eventstreams: move kafka config to new ca-bundle
operations/deployment-chartsmaster+9 -4helmfile.d: move eventgate* to the WMF CA cert bundle
operations/puppetproduction+4 -13varnishkafka: use new ca bundle instead of the Puppet one
mediawiki/services/eventstreamsmaster+1 -1blubber: deploy the wmf-certificates package in prod
eventgate-wikimediamaster+1 -1blubber: add wmf-certificates to the Docker images
operations/puppetproduction+1 -1netflow: move kafka config to new CA bundle
operations/puppetproduction+1 -1atskafka: use the same ca certificate as varnishkafka
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Nov 19 2021, 8:24 AM
Restricted Application added a project: Analytics. · View Herald TranscriptNov 19 2021, 8:24 AM
odimitrijevic triaged this task as Medium priority.Nov 22 2021, 5:09 PM
odimitrijevic edited projects, added Analytics-Radar; removed Analytics.
elukey changed the task status from Open to Stalled.Nov 24 2021, 3:59 PM

Setting this to stalled until we agree on https://phabricator.wikimedia.org/T296089

gerritbot added a comment.Nov 30 2021, 7:44 AM

Change 742671 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] atskafka: use the same ca certificate as varnishkafka

https://gerrit.wikimedia.org/r/742671

gerritbot added a project: Patch-For-Review.Nov 30 2021, 7:44 AM
gerritbot added a comment.Nov 30 2021, 7:46 AM

Change 742671 merged by Elukey:

[operations/puppet@production] atskafka: use the same ca certificate as varnishkafka

https://gerrit.wikimedia.org/r/742671

Maintenance_bot removed a project: Patch-For-Review.Nov 30 2021, 8:10 AM
gerritbot added a comment.Nov 30 2021, 3:32 PM

Change 742747 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] varnishkafka: use new ca bundle instead of the Puppet one

https://gerrit.wikimedia.org/r/742747

gerritbot added a project: Patch-For-Review.Nov 30 2021, 3:32 PM
elukey changed the task status from Stalled to In Progress.Nov 30 2021, 3:40 PM
gerritbot added a comment.Nov 30 2021, 3:49 PM

Change 742753 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] netflow: move kafka config to new CA bundle

https://gerrit.wikimedia.org/r/742753

elukey added a subscriber: Jgreen.Nov 30 2021, 4:21 PM

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

Jgreen added a comment.Nov 30 2021, 4:50 PM
In T296064#7537963, @elukey wrote:

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

Hey @elukey, this should not be a problem however this is exactly the wrong time of year to mess with the kafkatee pipeline. Can we postpone until early January?

elukey added a comment.Nov 30 2021, 5:29 PM
In T296064#7538028, @Jgreen wrote:
In T296064#7537963, @elukey wrote:

@Jgreen Hi! I am trying to move the Kafka Jumbo brokers TLS certs to the new PKI Intermediate CA dedicated to them, that will finally allow us to have per-host TLS certificates and stop using the Puppet CA. Before doing any switch all clients needs to trust the Root PKI CA cert and the Puppet CA one, so that I'll be able to move one broker at the time without impacting clients.

The client TLS certificates for the moment will not be touched.

We have created some helper functions and puppet code in profile::base::certificates, for prod we are basically using what's provided by the package wmf-certificates (that provides /etc/ssl/certs/wmf-ca-certificates.crt). I am not familiar with the code that you run on Fundraising, let me know if it is feasible to move the kafkatee's config to the new bundle on your side.

More info in T296089#7537901

Thanks in advance!

Hey @elukey, this should not be a problem however this is exactly the wrong time of year to mess with the kafkatee pipeline. Can we postpone until early January?

Sure makes sense, we can postpone it. I'll try to work on other clusters before Jumbo :)

elukey changed the task status from In Progress to Stalled.Nov 30 2021, 5:30 PM

Back to stalled, let's do it in January!

BTullis mentioned this in T296990: Move kafka-jumbo to a fixed uid/gid.Dec 6 2021, 9:55 AM
gerritbot added a comment.Dec 6 2021, 2:44 PM

Change 742753 merged by Elukey:

[operations/puppet@production] netflow: move kafka config to new CA bundle

https://gerrit.wikimedia.org/r/742753

odimitrijevic added a project: Data-Engineering-Radar.Jan 5 2022, 8:52 PM
Restricted Application removed a project: Data-Engineering. · View Herald TranscriptJan 5 2022, 8:52 PM
Jgreen closed subtask T296765: puppetize CA changes for kafkatee on fundraising banner loggers as Resolved.Jan 10 2022, 5:54 PM
elukey changed the task status from Stalled to In Progress.Jan 11 2022, 8:21 AM

Back to in-progress, the FR kafkatee instances moved to the new bundle!

gerritbot added a comment.Jan 11 2022, 8:39 AM

Change 752992 had a related patch set uploaded (by Elukey; author: Elukey):

[eventgate-wikimedia@master] blubber: add wmf-certificates to the Docker images

https://gerrit.wikimedia.org/r/752992

elukey added a comment.EditedJan 11 2022, 8:42 AM

Next steps:

gerritbot added a comment.Jan 11 2022, 2:23 PM

Change 752992 merged by Ottomata:

[eventgate-wikimedia@master] blubber: add wmf-certificates to the Docker images

https://gerrit.wikimedia.org/r/752992

gerritbot added a comment.Jan 12 2022, 9:38 AM

Change 753425 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: move eventgate-analytics* to the WMF CA cert bundle

https://gerrit.wikimedia.org/r/753425

gerritbot added a comment.Jan 12 2022, 9:47 AM

Change 753428 had a related patch set uploaded (by Elukey; author: Elukey):

[mediawiki/services/eventstreams@master] blubber: deploy the wmf-certificates package in prod

https://gerrit.wikimedia.org/r/753428

gerritbot added a comment.Jan 20 2022, 9:40 AM

Change 753428 merged by Elukey:

[mediawiki/services/eventstreams@master] blubber: deploy the wmf-certificates package in prod

https://gerrit.wikimedia.org/r/753428

Stashbot added a comment.Jan 26 2022, 10:07 AM

Mentioned in SAL (#wikimedia-analytics) [2022-01-26T10:07:27Z] <btullis> btullis@cumin1001:~$ sudo cumin 'O:cache::upload or O:cache::text' 'disable-puppet btullis-T296064-T299401'

gerritbot added a comment.Jan 26 2022, 10:12 AM

Change 742747 merged by Btullis:

[operations/puppet@production] varnishkafka: use new ca bundle instead of the Puppet one

https://gerrit.wikimedia.org/r/742747

elukey added a comment.Jan 26 2022, 11:00 AM

The last clients to move should be eventstreams and eventgate!

Next steps:

gerritbot added a comment.Jan 26 2022, 2:37 PM

Change 753425 merged by Elukey:

[operations/deployment-charts@master] helmfile.d: move eventgate* to the WMF CA cert bundle

https://gerrit.wikimedia.org/r/753425

Stashbot added a comment.Jan 26 2022, 2:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T14:41:39Z] <ottomata> deploying new CA certs for all eventgate services... T296064

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2022, 3:10 PM
Stashbot added a comment.Jan 26 2022, 3:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T15:24:45Z] <ottomata> paused (for meetings) in deploying new CA certs for all eventgate services, still TODO: eventgate-analytics-external, eventgate-main - T296064

Stashbot added a comment.Jan 27 2022, 2:54 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T14:54:15Z] <ottomata> continuing deployments of eventgate-main and eventgate-analytics to pick up CA cert changes - T296064 (also deploying eventgate-main for a schema repo bump for search)

gerritbot added a comment.Jan 27 2022, 3:20 PM

Change 757672 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] eventstreams: move kafka config to new ca-bundle

https://gerrit.wikimedia.org/r/757672

gerritbot added a project: Patch-For-Review.Jan 27 2022, 3:20 PM
gerritbot added a comment.Jan 27 2022, 3:37 PM

Change 757672 merged by Ottomata:

[operations/deployment-charts@master] eventstreams: move kafka config to new ca-bundle

https://gerrit.wikimedia.org/r/757672

elukey added a comment.Jan 28 2022, 7:05 AM

From what I can see from netstat on Jumbo nodes, all the clients that may be affected by this transition have been ported to the new CA bundle. This means that we could move one broker to the new PKI once we feel it is the right time.

Question mark about what to do in deployment prep, since IIRC there is kafka cluster in there using TLS certs.

elukey added a comment.Sep 14 2022, 2:33 PM

Getting back to the task - we have moved kafka logging-codfw to PKI (eqiad will follow soon), so all the upgrade workflow is sound and safe (we had only tested it on kafka test).

Next steps:

  • Think about deployment-prep and move the hosts to PKI in there too.
  • Move Kafka Jumbo to PKI

Is it something that the DE team can work on during the next months?

Ottomata added subscribers: odimitrijevic, emil.Sep 20 2022, 2:30 PM

Ping @BTullis @odimitrijevic @emil

BTW, @elukey I will likely reach out to you about how to do PKI right from the get go for T314156: Q1:rack/setup/install kafka-stretch100[12]

elukey added a comment.EditedSep 20 2022, 5:30 PM
In T296064#8248225, @Ottomata wrote:

BTW, @elukey I will likely reach out to you about how to do PKI right from the get go for T314156: Q1:rack/setup/install kafka-stretch100[12]

Sure! In theory all that is needed is:

profile::kafka::broker::ssl_generate_certificates: true
profile::base::certificates::include_bundle_jks: true

And the keystore password set in Puppet private (as we currently do for the puppet-based certs).

Do you think that we could try to test kafka 2.x or 3.x for the new cluster? Or would it derail/slowdown too much? I can help of course!

Ottomata added a comment.Sep 20 2022, 5:32 PM

Oh yeah, and hopefully even KRaft. More context in T307944: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP

elukey added a comment.Oct 5 2022, 7:10 AM

I added a detailed plan for kafka-main in T319372 :)

lbowmaker moved this task from Backlog to Investigate on the Event-Platform Value Stream board.Nov 16 2022, 4:43 PM
Ottomata mentioned this in T323771: Update varnishkafka client certificate for authenticating to kafka-jumbo.Dec 7 2022, 3:38 PM
Ottomata removed a project: Patch-For-Review.Jan 11 2023, 3:16 PM
Ottomata moved this task from Investigate to Backlog on the Event-Platform Value Stream board.Jan 11 2023, 3:32 PM
gerritbot added a comment.Tue, Mar 21, 10:39 AM

Change 901549 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::kafka::jumbo::broker: enable PKI migration settings

https://gerrit.wikimedia.org/r/901549

gerritbot added a project: Patch-For-Review.Tue, Mar 21, 10:39 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL