Tune Kafka logs to register clients connected
Open, NormalPublic


Our Kafka alarms are currently not offering any way of figuring out what clients (producers/consumers) are connected and what is their IP address.

In T172681 this would have been really useful to trace the faulty producer back to rhenium.wikimedia.org, rather than having to restart a broker with verbose logging.

Since we are introducing Kafka ACLs with the new Jumbo cluster we could simply tune the kafka-authorizer.log (I did it in labs when testing and it was quite handy).

elukey created this task.Aug 17 2017, 10:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2017, 10:09 AM
mforns moved this task from Incoming to Q1 (July 2017) on the Analytics board.Aug 17 2017, 3:19 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.
elukey edited projects, added Analytics-Kanban; removed Analytics.Tue, Sep 5, 1:23 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey added a comment.Tue, Sep 5, 1:53 PM

Tuning the kafka-authorizer appender is definitely important for us since it contains interesting info like:

[2017-09-05 13:39:32,147] DEBUG Principal = User:ANONYMOUS is Denied Operation = Describe from host = on resource = Topic:__confluent.support.metrics (kafka.authorizer.logger)
[2017-09-05 13:50:59,698] DEBUG operation = Describe on resource = Topic:elukey2 from host = is Allow based on acl = User:CN=client1,OU=Services,O=WMF,C=US has Allow permission for operations: Describe from hosts: * (kafka.authorizer.logger)
[2017-09-05 13:50:59,698] DEBUG Principal = User:CN=client1,OU=Services,O=WMF,C=US is Allowed Operation = Describe from host = on resource = Topic:elukey2 (kafka.authorizer.logger)

It doesn't show more detailed information about the kafka client (like api-version used, etc..) but the most important ones are there, like IP address and type of operation.

Change 376015 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] confluent::kafka: set kafka-authorizer log to DEBUG


Mentioned in SAL (#wikimedia-operations) [2017-09-06T11:24:57Z] <elukey> temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493

Change 376015 abandoned by Elukey:
confluent::kafka: set kafka-authorizer log to DEBUG

The patch didn't work on Kafka analytics since we need to enable a parameter to allow ACLs to be processed before getting any data on the authorizer.log. I'll try to come up with a new patch for the kafka jumbo cluster.