Page MenuHomePhabricator

Port varnishlog consumers to log to syslog / logging infra
Closed, ResolvedPublic

Description

ATM all daemons that read from varnishlog log directly to logstash inputs, they should do local syslog instead (prefixing json with @cee) and can then be ingested into the logging pipeline.

modules/varnish/files/varnishfetcherr.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishospital.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishslowlog.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer
modules/varnish/files/varnishtlsinspector.py:from wikimedia_varnishlogconsumer import BaseVarnishLogConsumer

Details

Related Gerrit Patches:

Event Timeline

fgiunchedi updated the task description. (Show Details)Jul 2 2019, 4:11 PM
MoritzMuehlenhoff triaged this task as Medium priority.Jul 3 2019, 8:18 AM

Change 563430 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: format log consumer stdout as cee+json

https://gerrit.wikimedia.org/r/563430

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 13 2020, 10:08 AM

Change 563977 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: use syslog for varnishlog consumers

https://gerrit.wikimedia.org/r/563977

ema added a subscriber: ema.
ema moved this task from Triage to Caching on the Traffic board.Jan 15 2020, 12:29 PM

Change 563430 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: format log consumer stdout as cee+json

https://gerrit.wikimedia.org/r/563430

Change 563977 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/563977

Had to revert in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569529, at least two issues found:

  1. journald < buster has a maximum line length of 2k, thus long lines get broken into multiple lines, in turn breaking json parsing.
  2. log records from *varnish* are sent also to plain syslog because of a custom rsyslog rule:
# cat /etc/rsyslog.d/80-varnish.conf 
# rsyslog configuration for Varnish

# This file is managed by Puppet

# Forward Varnish logs to logstash
if $programname contains "varnish" then {
    @logstash.svc.eqiad.wmnet:10514
}

Change 569533 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: switch logs from syslog to logging pipeline

https://gerrit.wikimedia.org/r/569533

Change 569533 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: switch logs from syslog to logging pipeline

https://gerrit.wikimedia.org/r/569533

Had to revert in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569529, at least two issues found:

  1. journald < buster has a maximum line length of 2k, thus long lines get broken into multiple lines, in turn breaking json parsing.

To fix this we can deploy a localhost syslog udp endpoint which doesn't suffer from this limitation, or (IMHO better) wait for Buster on cache cluster (T242093)

  1. log records from *varnish* are sent also to plain syslog because of a custom rsyslog rule:

Fixed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/569533, now all *varnish* programs are logged to kafka instead

Change 572012 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/572012

Change 572012 merged by Filippo Giunchedi:
[operations/puppet@production] varnish: use journald for varnishlog consumers

https://gerrit.wikimedia.org/r/572012

fgiunchedi closed this task as Resolved.Tue, Feb 18, 1:40 PM
fgiunchedi claimed this task.

This is now complete! All varnish logging goes through the logging pipeline