Move varnishkafka to PKI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	May 31 2023, 8:37 AM

Description

This is similar to T337248, it would be great to have varnishkafka instances on cache nodes to use PKI-based TLS certificates when connecting to Kafka brokers. At the moment we use a cergen certificate, that has CN: varnishkafka, used by Kafka brokers as username when evaluating ACLs (for example, only varnishkafka can produce messages to webrequest topics).

I filed 3 code reviews starting from https://gerrit.wikimedia.org/r/c/operations/puppet/+/924506, to do the following:

Create a catch-all systemd unit called varnishkafka-all that restarts all the varnishkafka instances present on a cache node.
Add the possibility to provision a PKI TLS certificate with CN:varnishkafka on cache nodes.
Apply the settings to cp4037 (depool the node, send some test traffic, check error logs and if msgs are landing to Kafka, etc..).

If the above works we could then apply the setting to all cache nodes incrementally. Lemme know your thoughts!

Details

Subject	Repo	Branch	Lines +/-
Simplify profile::cache::kafka::certificate to only support PKI/cfssl	operations/puppet	production	+13 -94
Move esams varnishkafka instances to PKI	operations/puppet	production	+8 -34
Move drmrs Varnishkafka instances to PKI	operations/puppet	production	+6 -0
Move eqiad varnishkafka instances to PKI	operations/puppet	production	+6 -0
role::cache::{text,upload}: move vk codfw instances to PKI	operations/puppet	production	+6 -0
role::cache::{text,upload}: move vk instances to PKI in eqsin	operations/puppet	production	+8 -0
role::cache::{text,upload}: move ulsfo varnishkafkas to PKI	operations/puppet	production	+8 -3
Move varnishkafka instances on cp4037 to PKI	operations/puppet	production	+3 -0
Move cp4037's varnishkafka instances to PKI	operations/puppet	production	+2 -0
profile::cache::kafka::certificate: fix client PKI config	operations/puppet	production	+7 -20
Move cp4037's varnishkafka instances to PKI	operations/puppet	production	+1 -0
profile::cache::kafka::certificate: use root instead of the kafka user	operations/puppet	production	+4 -4
profile::cache::kafka: add support for PKI	operations/puppet	production	+146 -75
varnishkafka: add catch all systemd unit	operations/puppet	production	+17 -1

Related Objects

Mentioned Here: T337248: kafka_mirror_maker TLS cert about to expire - 2023

Event Timeline

elukey created this task.May 31 2023, 8:37 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2023, 8:37 AM

elukey added a project: Data-Platform-SRE.May 31 2023, 8:37 AM

Maintenance_bot added a project: SRE.May 31 2023, 8:45 AM

Change 924506 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] varnishkafka: add catch all systemd unit

https://gerrit.wikimedia.org/r/924506

Change 924507 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::kafka: add support for PKI

https://gerrit.wikimedia.org/r/924507

Change 924509 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move cp4037's varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/924509

Change 924506 merged by Elukey:

[operations/puppet@production] varnishkafka: add catch all systemd unit

https://gerrit.wikimedia.org/r/924506

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:23:09Z] <elukey> all varnishkafka instances on caching nodes are getting restarted due to https://gerrit.wikimedia.org/r/c/operations/puppet/+/928087 - T337825

Mentioned in SAL (#wikimedia-analytics) [2023-06-07T15:23:12Z] <elukey> all varnishkafka instances on caching nodes are getting restarted due to https://gerrit.wikimedia.org/r/c/operations/puppet/+/928087 - T337825

The new varnishkafka-all unit is being rolled out across all cp nodes.

Next steps:

Merge https://gerrit.wikimedia.org/r/924507 (no-op, just adding the support to puppet)
Depool cp4037 and apply https://gerrit.wikimedia.org/r/924509

Change 924507 merged by Elukey:

[operations/puppet@production] profile::cache::kafka: add support for PKI

https://gerrit.wikimedia.org/r/924507

Fabfur subscribed.Jun 9 2023, 11:02 AM

Change 924509 merged by Elukey:

[operations/puppet@production] Move cp4037's varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/924509

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2023, 1:30 PM

Change 928846 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::kafka::certificate: use root instead of the kafka user

https://gerrit.wikimedia.org/r/928846

gerritbot added a project: Patch-For-Review.Jun 9 2023, 1:31 PM

Change 928846 merged by Elukey:

[operations/puppet@production] profile::cache::kafka::certificate: use root instead of the kafka user

https://gerrit.wikimedia.org/r/928846

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2023, 2:10 PM

Jun 09 14:05:42 cp4037 varnishkafka[3568251]: %3|1686319542.526|FAIL|varnishkafka#producer-1| [thrd:ssl://kafka-jumbo1009.eqiad.wmnet:9093/bootstrap]: ssl://kafka-jumbo1009.eqiad.wmnet:9093/bootstrap: SSL handshake failed: ../ssl/statem/statem_clnt.c:393: error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message: client SSL authentication might be required (see ssl.key.location and ssl.certificate.location and consult the broker logs for more information) (after 221ms in state CONNECT, 8 identical error(s) suppressed)

The above happened on cp4037 (depooled). After seeing https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka#configure-librdkafka-clientvar I have the suspicion that librdkafka wants a .pem file, or a key file, not a keystore.

Change 928854 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::kafka::certificate: fix client PKI config

https://gerrit.wikimedia.org/r/928854

gerritbot added a project: Patch-For-Review.Jun 9 2023, 2:29 PM

Change 928862 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move cp4037's varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/928862

Change 928854 merged by Elukey:

[operations/puppet@production] profile::cache::kafka::certificate: fix client PKI config

https://gerrit.wikimedia.org/r/928854

Change 928862 merged by Elukey:

[operations/puppet@production] Move cp4037's varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/928862

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2023, 8:30 AM

I've replicated a successful mTLS handshake with openssl s_client using the following CMD:

vgutierrez@cp4037:~$ sudo openssl s_client -connect kafka-jumbo1001.eqiad.wmnet:9093  -cert /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11.pem -key /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11-key.pem -curves prime256v1 -cipher ECDHE-ECDSA-AES256-GCM-SHA384 -cert_chain /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11.chain.pem -sigalgs ECDSA+SHA256

the problem here is that librdkafka doesn't have an analogue version to -cert_chain but it attempts to load the file from the same PEM file that contains the cert file:

if (rk->rk_conf.ssl.cert_location) {
        rd_kafka_dbg(rk, SECURITY, "SSL",
                     "Loading public key from file %s",
                     rk->rk_conf.ssl.cert_location);

        r = SSL_CTX_use_certificate_chain_file(
            ctx, rk->rk_conf.ssl.cert_location);

        if (r != 1) {
                rd_snprintf(errstr, errstr_size,
                            "ssl.certificate.location failed: ");
                return -1;
        }
}

SSL_CTX_use_certificate_chain_file() documentation says:

SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA

must be sorted starting with the subject's certificate BUT kafka__varnishkafka_kafka_11.chained.pem starts with the intermediate CA one rather than the subject's certificate as expected by OpenSSL

Change 929619 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move varnishkafka instances on cp4037 to PKI

https://gerrit.wikimedia.org/r/929619

gerritbot added a project: Patch-For-Review.Jun 13 2023, 6:48 AM

Change 929619 merged by Elukey:

[operations/puppet@production] Move varnishkafka instances on cp4037 to PKI

https://gerrit.wikimedia.org/r/929619

Mentioned in SAL (#wikimedia-operations) [2023-06-13T07:10:08Z] <elukey> move varnishkafka instances on cp4037 to PKI TLS certs - T337825

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2023, 7:10 AM

All vk instances running on cp4037, next steps:

Monitor cp4037 to verify that nothing explodes.
Extend the change to ulsfo and monitor.
Extend the change to all other DCs and monitor.
Clean up the old cert (Puppet master, Puppet CA, nodes, etc..)

Change 929963 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::cache::{text,upload}: move ulsfo varnishkafkas to PKI

https://gerrit.wikimedia.org/r/929963

gerritbot added a project: Patch-For-Review.Jun 14 2023, 7:59 AM

Change 929963 merged by Elukey:

[operations/puppet@production] role::cache::{text,upload}: move ulsfo varnishkafkas to PKI

https://gerrit.wikimedia.org/r/929963

Mentioned in SAL (#wikimedia-operations) [2023-06-15T09:05:40Z] <elukey> move varnishkafka instances in ulsfo to PKI - T337825

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2023, 9:10 AM

Next steps:

Roll out the changes to eqsin, and monitor.
Roll out the changes to codfw, and monitor.
Roll out the changes to eqiad, and monitor.
Roll out the changes to esams, and monitor.
Clean up the old cert (Puppet master, Puppet CA, nodes, etc..)

Change 930633 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::cache::{text,upload}: move vk instances to PKI in eqsin

https://gerrit.wikimedia.org/r/930633

gerritbot added a project: Patch-For-Review.Jun 15 2023, 1:42 PM

Change 930633 merged by Elukey:

[operations/puppet@production] role::cache::{text,upload}: move vk instances to PKI in eqsin

https://gerrit.wikimedia.org/r/930633

Mentioned in SAL (#wikimedia-analytics) [2023-06-19T14:04:43Z] <elukey> move varnishafka instances in eqsin to PKI - T337825

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2023, 2:11 PM

Change 931498 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::cache::{text,upload}: move vk instances to PKI

https://gerrit.wikimedia.org/r/931498

gerritbot added a project: Patch-For-Review.Jun 20 2023, 6:52 AM

Change 931498 merged by Elukey:

[operations/puppet@production] role::cache::{text,upload}: move vk codfw instances to PKI

https://gerrit.wikimedia.org/r/931498

Mentioned in SAL (#wikimedia-analytics) [2023-06-21T12:51:39Z] <elukey> move varnishafka instances in codfw to PKI - T337825

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2023, 1:10 PM

Change 932217 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move eqiad varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932217

Change 932218 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move esams varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932218

Change 932219 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move drmrs Varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932219

Change 932217 merged by Elukey:

[operations/puppet@production] Move eqiad varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932217

Mentioned in SAL (#wikimedia-analytics) [2023-06-22T13:16:57Z] <elukey> move varnishafka instances in eqiad to PKI - T337825

Change 932219 merged by Elukey:

[operations/puppet@production] Move drmrs Varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932219

Mentioned in SAL (#wikimedia-analytics) [2023-06-23T12:40:38Z] <elukey> move varnishkafka drmrs instances to pki - T337825

Change 932218 merged by Elukey:

[operations/puppet@production] Move esams varnishkafka instances to PKI

https://gerrit.wikimedia.org/r/932218

Mentioned in SAL (#wikimedia-analytics) [2023-06-26T14:06:01Z] <elukey> move varnishkafka instances in esams to pki - T337825

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 2:10 PM

All varnishkafkas on PKI!

Remaining steps:

clean up the old certificate from puppet private and puppet CA.

Mentioned in SAL (#wikimedia-operations) [2023-06-27T08:38:42Z] <elukey> revoked puppet cert for 'varnishkafka' and cleaned up its cergen's files in puppet private - T337825

elukey closed this task as Resolved.Jun 27 2023, 8:39 AM

elukey claimed this task.

Gehel moved this task from Incoming to Done on the Data-Platform-SRE board.Jul 19 2023, 8:56 AM

Change #1030052 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Simplify profile::cache::kafka::certificate to only support PKI/cfssl

https://gerrit.wikimedia.org/r/1030052

gerritbot added a project: Patch-For-Review.May 10 2024, 9:58 AM