We have several python daemons that collect derived metrics by reading and parsing output from varnishstat via the CacheStats class. The same metrics need to be exported to Prometheus too.
Daemons to work on:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T177195 Reduce technical debt in metrics monitoring | |||
| Resolved | fgiunchedi | T177199 Add Prometheus client support for varnish/statsd metrics daemons | |||
| Resolved | fgiunchedi | T183926 Limit http methods reported by varnishmtail | |||
| Resolved | fgiunchedi | T184942 Deprecate python varnish cachestats | |||
| Resolved | Krinkle | T190978 Update ResourceLoader dashboard to query varnishrls data from Prometheus instead |
IMO we could approach the problem of getting the stats above to Prometheus in at least two ways:
Going with 2. would mean adding another daemon and having to write a map statsd -> prometheus, at that point we might as well instrument the code. Given that we (ops/traffic) control the code and overall it isn't a whole lot of metrics approach 1. is preferable.
Change 387817 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] VCL: add layer information to X-Cache-Status
Change 388064 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] VCL: log TLS information to VSM
Change 387817 merged by Ema:
[operations/puppet@production] VCL: add layer information to X-Cache-Status
The approach we have in mind can roughly be summed up with varnishncsa | mtail. We can group the six scripts to be ported into frontend and backend ones:
Frontend
Backend
Let's focus on the frontend part first. Once TLS info logging to VSM is merged, in (very)pseudo-mtail, varnishxcache, varnishxcps and varnishmedia would be something like:
varnishncsa -n frontend -q 'ReqMethod ne "PURGE"' -F 'cache_status=%{X-Cache-Status}o\nkey_exchange=%{VCL_Log:CP-Key-Exchange}x\nhttp2=%{VCL_Log:CP-HTTP2}x\nURL=%U http_status=%s ' | mtail '
/^cache_status/ { xcache[X-Cache-Status]++ }
/^key_exchange/ { key_exchange[CP-Key-Exchange]++ }
[...]
/^URL=\/thumb\// { thumbnail[Http-Status]++ }
'varnishrls does some more funky stuff but in the end the fields used are %s - %{Cache-Control}o - %{If-None-Match}i, so I suspect it's either not going to be outrageously hard to write it in mtail, or we can easily embed the logic in VCL and use std.log as we've done for varnishxcps.
The remaining backend-related script, varnishstatsd, will need a separate varnishncsa process (the backend is another varnish instance with its own VSM!) and some more thought.
Change 394543 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] WIP: varnish: prometheus equivalent of statsd metrics daemons
Change 394597 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] mtail: add varnishmtail tests
Change 394543 merged by Ema:
[operations/puppet@production] varnish: prometheus equivalent of statsd metrics daemons
Change 388064 merged by Ema:
[operations/puppet@production] VCL: log TLS information to VSM
Change 394597 merged by Ema:
[operations/puppet@production] mtail: add varnishmtail tests
Change 395578 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] mtail: port varnishxcps
Change 397774 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add mtail to varnish jobs
Change 397831 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: install varnishxcps.mtail
Change 397831 merged by Ema:
[operations/puppet@production] cache: install varnishxcps.mtail
Change 397774 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add mtail to varnish jobs
Change 397851 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add mtail to varnish-upload job
Change 397876 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnishxcps.mtail: use prometheus labels
Change 397851 merged by Ema:
[operations/puppet@production] prometheus: add mtail to varnish-upload job
Change 397889 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] mtail: restructure tests
Change 397889 merged by Filippo Giunchedi:
[operations/puppet@production] mtail: restructure tests
Change 397876 merged by Ema:
[operations/puppet@production] varnishxcps.mtail: use prometheus labels
Change 398441 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add xcps aggregation rules
Change 398441 merged by Ema:
[operations/puppet@production] prometheus: add xcps aggregation rules
Change 398819 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] mtail: add varnishreqstats.mtail
Change 398819 merged by Ema:
[operations/puppet@production] mtail: add varnishreqstats.mtail
Change 399199 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add reqstats aggregation rule
Change 399199 merged by Ema:
[operations/puppet@production] prometheus: add reqstats aggregation rule
Change 401516 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add mtail to varnish-text job
Change 401516 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add mtail to varnish-text job
Change 401526 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: add varnishmtail instance for varnish backends
Change 401535 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] mtail: add program to count varnish backend metrics
Change 401526 merged by Ema:
[operations/puppet@production] varnish: add varnishmtail instance for varnish backends
Change 401535 merged by Ema:
[operations/puppet@production] mtail: add program to count varnish backend metrics
Change 402022 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] mtail: update varnishbackend.mtail regex
Change 402022 merged by Ema:
[operations/puppet@production] mtail: update varnishbackend.mtail regex
Change 402055 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add backend varnish mtail job
Change 402055 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add backend varnish mtail job
Change 402328 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: aggregate varnish_requests rate
Change 402342 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnishmtail: specify reload action
Change 402353 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnishmtail: notify daemons upon mtail program modification
Change 402328 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: aggregate varnish_requests rate
Status update:
Change 402342 abandoned by Ema:
varnishmtail: specify reload action
Reason:
Not OK, sending HUP to the script (and to varnishncsa) causes a service restart.
Change 402353 merged by Ema:
[operations/puppet@production] varnishmtail: notify daemons upon mtail program modification