on Logging, pulling data from each other

Is this true? I'm not so sure, I thought they used ElasticSearch replication instead of Kafka replication.

In T250250#6058642, @Ottomata wrote:

on Logging, pulling data from each other

Is this true? I'm not so sure, I thought they used ElasticSearch replication instead of Kafka replication.

Didn't know it, removing..

Disabled puppet on jumbo1001, and added the following bits to consumer/producer properties:

security.protocol=SSL
ssl.truststore.location=/etc/kafka/ssl/truststore.jks
ssl.truststore.password=XXXXXXX
ssl.enabled.protocols=TLSv1.2
ssl.protocol=TLS
ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Everything seems to work fine. Two caveats:

• puppet needs a bit of a refactoring probably to share more between kafka and mirror maker, I'll see what it is feasible/worth.
• the trustore in this case includes the cert of the puppet CA, so it works fine with the TLS cert validation for kafka-main100x. If we wanted to do things properly we should include, for every mirror maker instance, the trustore of the target/mirrored kafka cluster (for example, to work if we were using separate self signed CAs). I don't think it is necessary to complicate the puppet code in this way, but mentioning anyway.

Change 593234 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::kafka::jumbo::broker: enable TLS for the mirror maker

https://gerrit.wikimedia.org/r/593234

Change 593234 merged by Elukey:
[operations/puppet@production] role::kafka::jumbo::broker: enable TLS for the mirror maker

https://gerrit.wikimedia.org/r/593234

Mentioned in SAL (#wikimedia-analytics) [2020-04-29T14:26:28Z] <elukey> enable TLS consumer/producers for kafka main -> jumbo mirror maker - T250250

Kafka main is already done, all good!

Ah! Since profile::kafka::mirror sets ssl.keystore.location, kafka-mirror will attempt to authenticate with Kafka. User:ANONYMOUS is allowed to do anything, but if a client authenticates and their principal does not have an ACL, it will fail. I just ran

kafka acls --add --allow-principal User:CN=kafka_mirror_maker --producer --topic '*'

To allow kafka_mirror_maker to produce to Kafka jumbo.

FYI I also had to change the profile::kafka::mirror::ssl_password in puppet private; the keystore deployed for mirror maker is the one that the client uses, not the one the broker uses, and it is encrypted with a different password, specified in modules/secret/secrets/certificates/certificate.manifests.d/kafka_mirror_maker.certs.yaml.

Sorry I didn't catch this before!

Also, puppet does not ensure mirror maker will restart on config change, which is why you didn't see this problem when you applied puppet; kafka-mirror was never restarted. It restarted on kafka-jumbo1006 today and failed due to these problems. Should we make kafka-mirror subscribe to its config files? Not sure.

Mentioned in SAL (#wikimedia-analytics) [2020-05-14T18:41:43Z] <ottomata> fixed TLS authentication for Kafka mirror maker on jumbo - T250250

In T250250#6138068, @Ottomata wrote:

Ah! Since profile::kafka::mirror sets ssl.keystore.location, kafka-mirror will attempt to authenticate with Kafka. User:ANONYMOUS is allowed to do anything, but if a client authenticates and their principal does not have an ACL, it will fail. I just ran

kafka acls --add --allow-principal User:CN=kafka_mirror_maker --producer --topic '*'

To allow kafka_mirror_maker to produce to Kafka jumbo.

This is really strange, I thought I had tested this explicitly restarting on kafka-jumbo1001 mirror maker, and I didn't see any issue in the logs. Really sorry, should have probably checked in a better way, my bad.

FYI I also had to change the profile::kafka::mirror::ssl_password in puppet private; the keystore deployed for mirror maker is the one that the client uses, not the one the broker uses, and it is encrypted with a different password, specified in modules/secret/secrets/certificates/certificate.manifests.d/kafka_mirror_maker.certs.yaml.

Again didn't catch this after restarting, my bad as well.

Sorry I didn't catch this before!

Well this is clearly my fault not yours :)

Also, puppet does not ensure mirror maker will restart on config change, which is why you didn't see this problem when you applied puppet; kafka-mirror was never restarted. It restarted on kafka-jumbo1006 today and failed due to these problems. Should we make kafka-mirror subscribe to its config files? Not sure.

For mirror maker it seems safe, it is not like restarting a broker, so it should be fine.