Port varnishlog consumers to log to syslog / logging infra
Closed, ResolvedPublic
Actions

Description

ATM all daemons that read from varnishlog log directly to logstash inputs, they should do local syslog instead (prefixing json with @cee) and can then be ingested into the logging pipeline.

modules/varnish/files/varnishfetcherr.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishospital.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishslowlog.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishtlsinspector.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer

Details

Related Gerrit Patches:

operations/puppet : production	varnish: use journald for varnishlog consumers
operations/puppet : production	varnish: switch logs from syslog to logging pipeline
operations/puppet : production	varnish: use journald for varnishlog consumers
operations/puppet : production	varnish: format log consumer stdout as cee+json

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T227080 Deprecate all non-Kafka logstash inputs
		Resolved		fgiunchedi	T227108 Port varnishlog consumers to log to syslog / logging infra

Event Timeline

fgiunchedi created this task.Jul 2 2019, 4:05 PM

fgiunchedi updated the task description. (Show Details)Jul 2 2019, 4:11 PM

MoritzMuehlenhoff triaged this task as Medium priority.Jul 3 2019, 8:18 AM

fgiunchedi moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Aug 5 2019, 2:33 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:29 PM

Change 563430 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: format log consumer stdout as cee+json

https://gerrit.wikimedia.org/r/563430

gerritbot added a project: Patch-For-Review.Jan 10 2020, 12:19 PM

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 13 2020, 10:08 AM

Change 563977 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: use syslog for varnishlog consumers

https://gerrit.wikimedia.org/r/563977

ema added a project: Traffic.Jan 14 2020, 3:21 PM

ema added a subscriber: ema.

ema moved this task from Triage to Caching on the Traffic board.Jan 15 2020, 12:29 PM

Change 563430 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: format log consumer stdout as cee+json

https://gerrit.wikimedia.org/r/563430

Change 563977 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/563977

Had to revert in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569529, at least two issues found:

journald < buster has a maximum line length of 2k, thus long lines get broken into multiple lines, in turn breaking json parsing.
log records from *varnish* are sent also to plain syslog because of a custom rsyslog rule:

# cat /etc/rsyslog.d/80-varnish.conf 
# rsyslog configuration for Varnish

# This file is managed by Puppet

# Forward Varnish logs to logstash
if $programname contains "varnish" then {
    @logstash.svc.eqiad.wmnet:10514
}

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 3, 11:10 AM

Change 569533 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: switch logs from syslog to logging pipeline

https://gerrit.wikimedia.org/r/569533

gerritbot added a project: Patch-For-Review.Mon, Feb 3, 11:14 AM

Change 569533 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: switch logs from syslog to logging pipeline

https://gerrit.wikimedia.org/r/569533

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 3, 4:10 PM

In T227108#5844341, @fgiunchedi wrote:

Had to revert in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569529, at least two issues found:

journald < buster has a maximum line length of 2k, thus long lines get broken into multiple lines, in turn breaking json parsing.

To fix this we can deploy a localhost syslog udp endpoint which doesn't suffer from this limitation, or (IMHO better) wait for Buster on cache cluster (T242093)

log records from *varnish* are sent also to plain syslog because of a custom rsyslog rule:

Fixed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569533, now all *varnish* programs are logged to kafka instead

Change 572012 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/572012

gerritbot added a project: Patch-For-Review.Thu, Feb 13, 3:42 PM

Change 572012 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/572012

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 18, 11:10 AM