While investigating some refine errors, I found this (unrelated) error vk's logs:
Nov 30 17:11:53 cp3034 varnishkafka[24713]: KAFKADR: Kafka message delivery error: Local: Required feature not supported by broker
Never seen it before, better checking it more carefully :)
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| profile::cache::kafka::alerts: set more sensitive thresholds | operations/puppet | production | +4 -4 |
It happened on the 30th on various cp hosts, and in most of the Jumbo brokers I can see something like the following (repeated multiple times and for different brokers):
[2018-11-30 17:11:28,060] WARN [ReplicaFetcher replicaId=1006, leaderId=1002, fetcherId=11] Error in response for fetch request (type=FetchRequest, replicaId=1006, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={webrequest_upload-18=(offset=38392692796, logStartOffset=37553271595, maxBytes=4194304)}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=631834237, epoch=109933831)) (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1002 was disconnected before the response was readIt seems a network blip happened from ~17:10 to ~17:15, both varnishkafka and kafka metrics showed some changes.
Change 478210 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::cache::kafka::alerts: set more sensitive thresholds
Change 478210 merged by Elukey:
[operations/puppet@production] profile::cache::kafka::alerts: set more sensitive thresholds
The issue seems a one time only, and the new alarms have been very reliable over the past months. Closing this task since there seems to be no action left, will re-open if necessary.