Since last week there's constant kafka lag observed for benthos@mw_accesslog_metrics.service on centrallog1002:
(note the consumer group uses - and the systemd unit name uses _)
There's also a stream of constant errors in the journal (which might or might not be related, I'm not sure)
Jun 05 22:04:29 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_by_endpoint_duration path=root.pipeline.processors.2 Jun 05 22:04:29 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_by_endpoint_duration path=root.pipeline.processors.2 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=response_size_bytes path=root.pipeline.processors.0 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_duration path=root.pipeline.processors.1 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=response_size_bytes path=root.pipeline.processors.0 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_by_endpoint_duration path=root.pipeline.processors.2 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=response_size_bytes path=root.pipeline.processors.0 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_duration path=root.pipeline.processors.1 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_duration path=root.pipeline.processors.1 Jun 05 22:04:34 centrallog1002 benthos@mw_accesslog_metrics[926]: level=error msg="Value interpolation error: expected number value, got null" @service=benthos label=requests_by_endpoint_duration path=root.pipeline.processors.2
@kamila would you mind taking a look at both of these ? thank you!
