Sometimes ChangeProp gets into the mode when it starts logging KafkaConsumer is not connected in a tight loop. The logs normally disappeare after a full restart. The service was operating normally though.
Sample log entry:
{
"_index": "logstash-2018.07.12",
"_type": "cpjobqueue",
"_id": "AWSNgQkkKHrb1-g-EgFk",
"_version": 1,
"_score": null,
"_source": {
"err_executor": "RetryExecutor",
"err_errno": -172,
"source_host": "10.192.0.33",
"err_levelPath": "error/consumer",
"level": "ERROR",
"err_rule_name": "low_traffic_jobs",
"line": "260",
"err_code": -172,
"pid": 9774,
"err_origin": "kafka",
"type": "cpjobqueue",
"message": "KafkaConsumer is not connected",
"version": "1.0",
"normalized_message": "KafkaConsumer is not connected",
"tags": [
"es",
"gelf",
"normalized_message_untrimmed"
],
"err_message": "KafkaConsumer is not connected",
"file": "/srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/error.js",
"@timestamp": "2018-07-12T07:59:59.974Z",
"err_name": "cpjobqueue",
"host": "scb2003",
"@version": "1",
"gelf_level": "3",
"err_stack": "Error: KafkaConsumer is not connected\n at Function.createLibrdkafkaError [as create] (/srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/error.js:260:10)\n at /srv/deployment/cpjobqueue/deploy-cache/revs/ba672a32e1d9642f5dad2008a6f05b8b48ee89e6/node_modules/node-rdkafka/lib/kafka-consumer.js:442:29"
},
"fields": {
"@timestamp": [
1531382399974
]
},
"sort": [
1531382399974
]
}I believe the problem is that driver can't properly reconnect automatically when it's been disconnected. We need to figure out whether it's a driver bug or it's something we should fix/workaround in our code.