Update kafka-jumbo certificates
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	BTullis
	Nov 23 2022, 11:56 AM

Description

We need to update the certificates for the kafka-jumbo cluster. Our monitoring system has alerted us to their upcoming expiry.

SSL WARNING - Certificate kafka_jumbo-eqiad_broker valid until 2022-12-04 14:47:46 +0000 (expires in 11 days

The process to update the certificates is described here: https://wikitech.wikimedia.org/wiki/Kafka/Administration#Kafka_Certificates

Since we need to take action quickly, I am expediting this ticket into the current #shared-data-infrastrcuture sprint. I will assign the ticket to myself (@BTullis) and carry out the work, with @Stevemunene shadowing.

Details

Other Assignee: Stevemunene

Related Objects

Mentioned Here: T323771: Update varnishkafka client certificate for authenticating to kafka-jumbo

Event Timeline

BTullis created this task.Nov 23 2022, 11:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 23 2022, 11:56 AM

BTullis triaged this task as High priority.Nov 23 2022, 11:57 AM

BTullis moved this task from Backlog to Shared Data Infra on the Data-Engineering-Planning board.

BTullis moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)) board.

BTullis updated Other Assignee, added: Stevemunene.

BTullis set the point value for this task to 1.

We have verified the date of expiry by using the following command:

btullis@stat1004:~$ openssl s_client -connect kafka-jumbo1001.eqiad.wmnet:9093 | openssl x509 -noout -dates
depth=1 CN = Puppet CA: palladium.eqiad.wmnet
verify return:1
depth=0 CN = kafka_jumbo-eqiad_broker
verify return:1
notBefore=Dec  4 14:47:46 2017 GMT
notAfter=Dec  4 14:47:46 2022 GMT

Verified the existing certificates' status on the puppetmaster:

root@puppetmaster1001:/srv/private/modules/secret/secrets/certificates/certificate.manifests.d# cergen -c 'kafka_jumbo.*' --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

Status of certificates ['kafka_jumbo-eqiad_broker']

Certificate(kafka_jumbo-eqiad_broker, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.private.pem: PRESENT (mtime: 2019-10-11T12:49:56.622059)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.public.pem: PRESENT (mtime: 2019-10-11T12:49:56.622059)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: PRESENT (mtime: 2019-10-11T12:49:56.622059)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/ca.crt.pem: PRESENT (mtime: 2019-12-10T14:26:41.848698)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.p12: PRESENT (mtime: 2019-10-11T12:49:56.622059)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.jks: PRESENT (mtime: 2019-10-11T12:49:56.622059)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/truststore.jks: PRESENT (mtime: 2019-10-11T12:49:56.622059)

Disabling puppet temporarily on all kafka-jumbo servers.

btullis@cumin2002:~$ sudo cumin A:kafka-jumbo "disable-puppet 'btullis renewing certificates T323697'"
9 hosts will be targeted:
kafka-jumbo[1001-1009].eqiad.wmnet
Ok to proceed on 9 hosts? Enter the number of affected hosts to confirm or "q" to quit 9
===== NO OUTPUT =====                                                                                                                                                                                              
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (9/9) [00:04<00:00,  2.18hosts/s]
FAIL |                                                                                                                                                                             |   0% (0/9) [00:04<?, ?hosts/s]
100.0% (9/9) success ratio (>= 100.0% threshold) for command: 'disable-puppet '...ficates T323697''.
100.0% (9/9) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Clean the certificate:

root@puppetmaster1001:/srv/private/modules/secret/secrets/certificates/certificate.manifests.d# puppet cert clean kafka_jumbo-eqiad_broker
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 3406
Notice: Removing file Puppet::SSL::Certificate kafka_jumbo-eqiad_broker at '/var/lib/puppet/server/ssl/ca/signed/kafka_jumbo-eqiad_broker.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_jumbo-eqiad_broker at '/var/lib/puppet/server/ssl/certs/kafka_jumbo-eqiad_broker.pem'

Destroy the certificate:

root@puppetmaster1001:/srv/private/modules/secret/secrets/certificates/certificate.manifests.d# puppet cert destroy kafka_jumbo-eqiad_broker
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 3406

Regenerating the certificatres with the --generate and --force flags added to the previous command.

root@puppetmaster1001:/srv/private/modules/secret/secrets/certificates/certificate.manifests.d# cergen --generate --force -c 'kafka_jumbo.*' --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2022-11-23 15:18:16,922 INFO     cergen                                   Generating certificates ['kafka_jumbo-eqiad_broker'] with force=True
2022-11-23 15:18:16,922 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating all files, force=True...
2022-11-23 15:18:16,924 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating certificate file
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
2022-11-23 15:18:18,414 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating CA certificate file
2022-11-23 15:18:18,414 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating PKCS12 keystore file
2022-11-23 15:18:18,791 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating Java keystore file
2022-11-23 15:18:19,754 INFO     Certificate(kafka_jumbo-eqiad_broker)    Importing PuppetCA(puppetmaster1001.eqiad.wmnet_8140) cert into Java keystore
2022-11-23 15:18:20,734 INFO     Certificate(kafka_jumbo-eqiad_broker)    Generating Java truststore file with CA certificate PuppetCA(puppetmaster1001.eqiad.wmnet_8140)

Status of certificates ['kafka_jumbo-eqiad_broker']

Certificate(kafka_jumbo-eqiad_broker, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.private.pem: PRESENT (mtime: 2022-11-23T15:18:16.922145)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.public.pem: PRESENT (mtime: 2022-11-23T15:18:16.922145)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: PRESENT (mtime: 2022-11-23T15:18:18.410143)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/ca.crt.pem: PRESENT (mtime: 2022-11-23T15:18:18.410143)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.p12: PRESENT (mtime: 2022-11-23T15:18:18.430143)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.jks: PRESENT (mtime: 2022-11-23T15:18:20.174141)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/truststore.jks: PRESENT (mtime: 2022-11-23T15:18:21.070140)

Now committing and rolling out those certificates by manually running puppet.

The certificates have been commited to the private repo and merged.
I have verified that rolling out the new certificiates to the brokers with pouppet does not trigger a service restart.

btullis@kafka-jumbo1001:/etc/kafka/ssl$ systemctl status kafka
● kafka.service - Kafka Broker
   Loaded: loaded (/lib/systemd/system/kafka.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-09-28 09:23:17 UTC; 1 months 25 days ago
 Main PID: 31573 (java)
    Tasks: 213 (limit: 4915)
   Memory: 58.1G
   CGroup: /system.slice/kafka.service
           └─31573 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms2g -Xmx2g -server -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPer

Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:172)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:718)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:70)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.Selector.doClose(Selector.java:746)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.Selector.close(Selector.java:734)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at org.apache.kafka.common.network.Selector.poll(Selector.java:424)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at kafka.network.Processor.poll(SocketServer.scala:628)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at kafka.network.Processor.run(SocketServer.scala:545)
Nov 23 15:30:48 kafka-jumbo1001 kafka-server-start[31573]:         at java.lang.Thread.run(Thread.java:750)

btullis@kafka-jumbo1001:/etc/kafka/ssl$ sudo puppet agent -t -v
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for kafka-jumbo1001.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(30bc961f0a) Muehlenhoff - Enable profile::auto_restarts::service for Turnilo'
Notice: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/truststore.jks]/content: 
Binary files /etc/kafka/ssl/truststore.jks and /tmp/puppet-file20221123-25733-ek9a4y differ

Info: Computing checksum on file /etc/kafka/ssl/truststore.jks
Info: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/truststore.jks]: Filebucketed /etc/kafka/ssl/truststore.jks to puppet with sum 44ee9c506355cc1c9e0fde4526af6a3c
Notice: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/truststore.jks]/content: content changed '{md5}44ee9c506355cc1c9e0fde4526af6a3c' to '{md5}6dcddd421d1ee5098390ebb58b4f0a05'
Notice: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks]/content: 
Binary files /etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks and /tmp/puppet-file20221123-25733-1hx5p69 differ

Info: Computing checksum on file /etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks
Info: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks]: Filebucketed /etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks to puppet with sum 3ac1b983726d8aa6825e40beae7e5f7f
Notice: /Stage[main]/Profile::Kafka::Broker/File[/etc/kafka/ssl/kafka_jumbo-eqiad_broker.keystore.jks]/content: content changed '{md5}3ac1b983726d8aa6825e40beae7e5f7f' to '{md5}e6f08d8896be093c844cd1fa8faf7787'
Notice: Applied catalog in 20.67 seconds

btullis@kafka-jumbo1001:/etc/kafka/ssl$ systemctl status kafka
● kafka.service - Kafka Broker
   Loaded: loaded (/lib/systemd/system/kafka.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-09-28 09:23:17 UTC; 1 months 25 days ago
 Main PID: 31573 (java)
    Tasks: 213 (limit: 4915)
   Memory: 58.1G
   CGroup: /system.slice/kafka.service
           └─31573 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms2g -Xmx2g -server -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPer

Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,451] INFO [Log partition=webrequest_text-3, dir=/srv/kafka/data] Incrementing log start offse
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,451] INFO Cleared earliest 0 entries from epoch cache based on passed offset 489467254373 lea
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,468] INFO [Log partition=webrequest_upload-3, dir=/srv/kafka/data] Found deletable segments w
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,468] INFO [Log partition=webrequest_upload-3, dir=/srv/kafka/data] Scheduling log segment [ba
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,468] INFO [Log partition=webrequest_upload-3, dir=/srv/kafka/data] Incrementing log start off
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,469] INFO Cleared earliest 0 entries from epoch cache based on passed offset 228972381477 lea
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,478] INFO [Log partition=webrequest_text-8, dir=/srv/kafka/data] Found deletable segments wit
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,478] INFO [Log partition=webrequest_text-8, dir=/srv/kafka/data] Scheduling log segment [base
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,478] INFO [Log partition=webrequest_text-8, dir=/srv/kafka/data] Incrementing log start offse
Nov 23 15:32:05 kafka-jumbo1001 kafka-server-start[31573]: [2022-11-23 15:32:05,479] INFO Cleared earliest 0 entries from epoch cache based on passed offset 489458009726 lea
lines 1-19/19 (END)

Mentioned in SAL (#wikimedia-analytics) [2022-11-23T15:38:30Z] <btullis> roll-restarting kafka-jumbo brokers to pick up new certificates. T323697

Now re-enabling puppet on the remaining 8 hosts and running puppet to pull down the certificates.

btullis@cumin2002:~$ sudo cumin A:kafka-jumbo "puppet agent --enable"
9 hosts will be targeted:
kafka-jumbo[1001-1009].eqiad.wmnet
Ok to proceed on 9 hosts? Enter the number of affected hosts to confirm or "q" to quit 9
===== NO OUTPUT =====                                                                                                                                                                                              
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (9/9) [00:02<00:00,  3.02hosts/s]
FAIL |                                                                                                                                                                             |   0% (0/9) [00:02<?, ?hosts/s]
100.0% (9/9) success ratio (>= 100.0% threshold) for command: 'puppet agent --enable'.
100.0% (9/9) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
btullis@cumin2002:~$ sudo cumin A:kafka-jumbo "run-puppet-agent"

The cookbook to restart the brokers is running, but we have verified that the first broker has already restarted with the new certificates, which is a good sign.

btullis@stat1004:~$ openssl s_client -connect kafka-jumbo1001.eqiad.wmnet:9093 | openssl x509 -noout -dates
depth=1 CN = Puppet CA: palladium.eqiad.wmnet
verify return:1
depth=0 CN = kafka_jumbo-eqiad_broker
verify return:1
notBefore=Nov 22 15:18:18 2022 GMT
notAfter=Nov 22 15:18:18 2027 GMT

Great work! I found this alert in icinga, under puppemaster1001 -> Puppet CA expired certs:

crit: kafka_broker_kafka-jumbo1001 kafka_broker_kafka-jumbo1002 kafka_broker_kafka-jumbo1003 kafka_broker_kafka-jumbo1004 kafka_broker_kafka-jumbo1005 kafka_broker_kafka-jumbo1006 kafka_jumbo_broker
warn: kafka_client_test1 varnishkafka

It is probably some leftover/garbage to clean up, but better to be sure :)

Thanks @elukey. I'm taking a look at this now.
It's interesting because I found this: https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate

...but the cookbook only operates on hostnames, whereas these are all CN values, it seems.
It's possible that they're not in use any more, but I'll have to act carefully to make sure.

The roll-restart of the kafka-jumbo brokers completed successfully, so that's good.

@BTullis I think that most of the above are certs that we don't use anymore, like:

elukey@puppetmaster1001:~$ sudo ls /var/lib/puppet/server/ssl/ca/signed | grep jumbo
kafka_broker_kafka-jumbo1001.pem
kafka_broker_kafka-jumbo1002.pem
kafka_broker_kafka-jumbo1003.pem
kafka_broker_kafka-jumbo1004.pem
kafka_broker_kafka-jumbo1005.pem
kafka_broker_kafka-jumbo1006.pem
kafka-jumbo1001.eqiad.wmnet.pem
kafka-jumbo1002.eqiad.wmnet.pem
kafka-jumbo1003.eqiad.wmnet.pem
kafka-jumbo1004.eqiad.wmnet.pem
kafka-jumbo1005.eqiad.wmnet.pem
kafka-jumbo1006.eqiad.wmnet.pem
kafka-jumbo1007.eqiad.wmnet.pem
kafka-jumbo1008.eqiad.wmnet.pem
kafka-jumbo1009.eqiad.wmnet.pem
kafka_jumbo_broker.pem
kafka_jumbo-eqiad_broker.pem

elukey@puppetmaster1001:~$ sudo openssl x509 -in /var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1001.pem -text | grep CN
        Issuer: CN = Puppet CA: palladium.eqiad.wmnet
        Subject: CN = kafka_broker_kafka-jumbo1001, C = US, L = San Francisco, O = Wikimedia Foundation, ST = CA

I don't recall any usage of a CN like kafka_broker_kafka-jumbo1001 in any of our configs, so I think that we'd need to clean up those certs (they are not the host level ones, something different made from cergen if I have to guess).

The only interesting one is varnishkafka though, since we use it for client TLS auth:

elukey@cp2029:~$ sudo cat /etc/varnishkafka/ssl/varnishkafka.crt.pem | openssl x509 -noout -dates 
notBefore=Dec 13 15:55:06 2017 GMT
notAfter=Dec 13 15:55:06 2022 GMT

Great, I was coming to that conclusion too.
The fact that it only lists six kafka jumbo brokers by name (kafka_broker_kafka_jumbo100[1-6]) suggested that these certificates might be old and unused, since we added kafka-jumbo1007 into service on May 18th 2020.

Also you've listed kafka_jumbo-eqiad_broker.pem - Just double-checking, that one we do use as well, don't we? I've just regenerated it above.

And yes, the varnishkafka one jumped out at me too.

Removed unused and expiring kafka_jumbo certificates.

btullis@puppetmaster1001:/var/lib/puppet/server/ssl/ca/signed$ sudo puppet cert clean kafka_broker_kafka-jumbo1001 kafka_broker_kafka-jumbo1002 kafka_broker_kafka-jumbo1003 kafka_broker_kafka-jumbo1004 kafka_broker_kafka-jumbo1005 kafka_broker_kafka-jumbo1006 kafka_jumbo_broker
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 3392
Notice: Revoked certificate with serial 3393
Notice: Revoked certificate with serial 3397
Notice: Revoked certificate with serial 3395
Notice: Revoked certificate with serial 3396
Notice: Revoked certificate with serial 3394
Notice: Revoked certificate with serial 3400
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1001 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1001.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1001 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1001.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1002 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1002.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1002 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1002.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1003 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1003.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1003 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1003.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1004 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1004.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1004 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1004.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1005 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1005.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1005 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1005.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1006 at '/var/lib/puppet/server/ssl/ca/signed/kafka_broker_kafka-jumbo1006.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_broker_kafka-jumbo1006 at '/var/lib/puppet/server/ssl/certs/kafka_broker_kafka-jumbo1006.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_jumbo_broker at '/var/lib/puppet/server/ssl/ca/signed/kafka_jumbo_broker.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_jumbo_broker at '/var/lib/puppet/server/ssl/certs/kafka_jumbo_broker.pem'

Mentioned in SAL (#wikimedia-operations) [2022-11-24T13:53:51Z] <btullis> Removed unused and expiring kafka_jumbo certificates. T323697

btullis@puppetmaster1001:/var/lib/puppet/server/ssl/ca/signed$ sudo puppet cert clean kafka_client_test1
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 3408
Notice: Removing file Puppet::SSL::Certificate kafka_client_test1 at '/var/lib/puppet/server/ssl/ca/signed/kafka_client_test1.pem'
Notice: Removing file Puppet::SSL::Certificate kafka_client_test1 at '/var/lib/puppet/server/ssl/certs/kafka_client_test1.pem'

I have confirmed the varnishkafka client certificate expiry date on a cp host.

btullis@cp1075:/etc/varnishkafka/ssl$ cat varnishkafka.crt.pem | openssl x509 -noout -dates
notBefore=Dec 13 15:55:06 2017 GMT
notAfter=Dec 13 15:55:06 2022 GMT

In T323697#8419896, @BTullis wrote:

Also you've listed kafka_jumbo-eqiad_broker.pem - Just double-checking, that one we do use as well, don't we? I've just regenerated it above.

Yes yes that one is still in use!

And yes, the varnishkafka one jumped out at me too.

We use it to authenticate varnishkafka to jumbo since only the varnishkafka user is allowed to push to the webrequest topics.. So the cert needs to be renewed as well :)

And yes, the varnishkafka one jumped out at me too.

We use it to authenticate varnishkafka to jumbo since only the varnishkafka user is allowed to push to the webrequest topics.. So the cert needs to be renewed as well :)

Thanks for confirming. I think that for the sake of completeness I'll make a new ticket for updating the varnishkafka client authentication certificates. I know that's closely related to the brokers, but our future selves might thank us.

BTullis moved this task from In Progress to Done on the Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)) board.Nov 24 2022, 2:17 PM

+1, the varnishkafka cert is another good candidate for PKI in my opinion, but very out of scope I know :)

I have created a new ticket for the varnishkafka certificate renewal here: T323771: Update varnishkafka client certificate for authenticating to kafka-jumbo
It might be a good one for @Stevemunene to work on, given that he shadowed me on this ticket.

BTullis closed this task as Resolved.Nov 24 2022, 5:15 PM

Update kafka-jumbo certificatesClosed, ResolvedPublic1 Estimated Story PointsActions

Description

Details

Related Objects

Event Timeline

Update kafka-jumbo certificates
Closed, ResolvedPublic1 Estimated Story Points
Actions