varnish: implement FetchError logging
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ema
	Jun 4 2019, 2:36 PM

Description

We currently have no way of knowing the cause of Varnish fetch errors other than observing them live with varnishlog -g request -q 'FetchError ne "Pass delivery abandoned"'

Keep track of FetchError causes by sending the logs to logstash.

See https://varnish-cache.org/tips/varnishlog/fetcherror.html and https://varnish-cache.org/docs/5.1/reference/vsl.html

Details

Subject	Repo	Branch	Lines +/-
varnishfetcherror: do not overwrite multiple tags	operations/puppet	production	+12 -2
varnishfetcherr: do send logs, handle interesting tags	operations/puppet	production	+16 -6
varnishlog: implement varnishfetcherr	operations/puppet	production	+88 -6

Customize query in gerrit

Related Objects

Mentioned In: T189333: Changing Kibana filters is ridiculously slow
T226375: Investigate esams text varnish backend fetch failures

Event Timeline

• ema created this task.Jun 4 2019, 2:36 PM

Restricted Application added a project: SRE. · View Herald TranscriptJun 4 2019, 2:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as Medium priority.Jun 4 2019, 2:36 PM

elukey subscribed.Jun 4 2019, 2:55 PM

• ema moved this task from Backlog to Caching on the Traffic board.Jun 6 2019, 3:24 PM

• ema renamed this task from varnish: add X-Fetch-Error response header to varnish: implement FetchError logging .Jun 17 2019, 7:44 AM

• ema updated the task description. (Show Details)

Change 517375 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnishlog: implement varnishfetcherr

https://gerrit.wikimedia.org/r/517375

gerritbot added a project: Patch-For-Review.Jun 17 2019, 7:54 AM

The initial plan of adding a synthetic header to varnish with the FetchError cause seems a little too complicated to implement. Send error logs to logstash instead.

Change 517375 merged by Ema:
[operations/puppet@production] varnishlog: implement varnishfetcherr

https://gerrit.wikimedia.org/r/517375

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2019, 8:10 AM

Change 517649 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnishfetcherr: do send logs, handle interesting tags

https://gerrit.wikimedia.org/r/517649

gerritbot added a project: Patch-For-Review.Jun 18 2019, 2:00 PM

Change 517649 merged by Ema:
[operations/puppet@production] varnishfetcherr: do send logs, handle interesting tags

https://gerrit.wikimedia.org/r/517649