Sometimes ChangeProp gets into the mode when it starts logging KafkaConsumer is not connected in a tight loop. The logs normally disappeare after a full restart. The service was operating normally though.
Sample log entry:
{ "_index": "logstash-2018.07.12", "_type": "cpjobqueue", "_id": "AWSNgQkkKHrb1-g-EgFk", "_version": 1, "_score": null, "_source": { "err_executor": "RetryExecutor", "err_errno": -172, "source_host": "10.192.0.33", "err_levelPath": "error/consumer", "level": "ERROR", "err_rule_name": "low_traffic_jobs", "line": "260", "err_code": -172, "pid": 9774, "err_origin": "kafka", "type": "cpjobqueue", "message": "KafkaConsumer is not connected", "version": "1.0", "normalized_message": "KafkaConsumer is not connected", "tags": [ "es", "gelf", "normalized_message_untrimmed" ], "err_message": "KafkaConsumer is not connected", "file": "/srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/error.js", "@timestamp": "2018-07-12T07:59:59.974Z", "err_name": "cpjobqueue", "host": "scb2003", "@version": "1", "gelf_level": "3", "err_stack": "Error: KafkaConsumer is not connected\n at Function.createLibrdkafkaError [as create] (/srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/error.js:260:10)\n at /srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/kafka-consumer.js:442:29" }, "fields": { "@timestamp": [ 1531382399974 ] }, "sort": [ 1531382399974 ] }
I believe the problem is that driver can't properly reconnect automatically when it's been disconnected. We need to figure out whether it's a driver bug or it's something we should fix/workaround in our code.