Page MenuHomePhabricator

varnishlog consumers http request/response logging field explosion
Closed, InvalidPublic

Description

At the moment varnishlog consumers (i.e. varnishfetcherr, varnishslowlog, varnishtlsinspector, varnishhospital) will log all request/response headers as individual fields as key: value. Since there's a lot of custom/user-controller headers in the wild and each header gets indexed in elasticsearch as a separate field, we've seen "fields explosion" problems where we hit elasticsearch's per-index limits. See also T180051 for more context on this problem.

A potential solution would be to have a whitelisted set of request/response headers to send as individual fields, and consolidate the rest into e.g. request_headers and response_headers fields.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I thought we did already address the issue with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/520425/. Evidently there's something wrong with that patch. To be continued!

ema triaged this task as Medium priority.Nov 12 2019, 3:18 PM
ema moved this task from Backlog to Caching on the Traffic board.

Ah! It is possible that's working as expected and I overlooked recent log entries, I'll see if I can find a recent example

Confirmed only a subset of headers are whitelisted for sending, closing