Current logging output looks like:
start:1582152620 uptime:3775380 purgers:2 recvd:7919253941 bad:0 filtered:0 Purger0: input:7919253941 failed:0 q_size:86873061 q_mem:14865036237 q_max_size:157669187 q_max_mem:28148024297 Purger1: input:7832380879 failed:0 q_size:2331 q_mem:718320 q_max_size:129967 q_max_mem:18856476
with separate stat lines for each purger thread. But the monitoring exporter script expects a single purger worker, and also expects slightly different names (e.g. queue_size).
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| update vhtcpd exporter to match reality | operations/puppet | production | +78 -36 |
I had a chat with the author to make sure we understand the meaning of the fields:
First line:
start: this is just the *nix timestamp of daemon start
uptime: seconds since the above. Not sure if either of these is very useful (it's nice to see restarts, but the rest of the numbers will reset and make it obvious anyways)
purgers: fixed for life, just counts how many purger lines will follow
recvd: Counter of all purges received over multicast input
bad: Counter, subset of the above that were somehow faulty
filtered: Counter, subset of un-bad ones which were filtered out by a commandline regex filter
Purger lines: (0 is the backend, 1 is the frontend)
input: Counter, valid inputs to the queue
failed: Counter, failed to deliver via HTTP (unparseable response, or [45]xx response, etc)
q_size: Gauge, current amount of purges stuck in queue
q_mem: Gauge, current memory consumed by purges stuck in queue
q_max_size: Maximum value of q_size since daemon start - no need to consume this if prometheus has historical stats anyways!
q_max_mem: Maximum value of q_mem since daemon start - no need to consume this if prometheus has historical stats anyways!
Note that each PurgerN has its own separate queue. Purger0's "input" should track fairly close to recvd-bad-filtered from the top line. Purged items move through the multiple queues serially; Purger1 doesn't receive an item until Purger0 has successfully processed it. The purgers also have configurable queuing delays (I believe all are set to 1 second on the frontend, which is Purger1).
We're probably going to add multiple purger connections to fan out the per-thread load from T241232 to help with T249325 . I've poked at that already, and the likely impact here is that the overall format will stay the same, but the Purger0 labels at the start of the lines will probably change, so don't rely on the PurgerN formatting of that leading label. I'm thinking right now that I'll give the purgers names in the config and then number them for the configured fanout factor, so we're likely to see something more like:
start:1582152620 uptime:3775380 purgers:5 recvd:7919253941 bad:0 filtered:0 Backend0: input:7919253941 failed:0 q_size:86873061 q_mem:14865036237 q_max_size:157669187 q_max_mem:28148024297 Backend1: input:7919253941 failed:0 q_size:86873061 q_mem:14865036237 q_max_size:157669187 q_max_mem:28148024297 Backend2: input:7919253941 failed:0 q_size:86873061 q_mem:14865036237 q_max_size:157669187 q_max_mem:28148024297 Backend3: input:7919253941 failed:0 q_size:86873061 q_mem:14865036237 q_max_size:157669187 q_max_mem:28148024297 Frontend0: input:7832380879 failed:0 q_size:2331 q_mem:718320 q_max_size:129967 q_max_mem:18856476
And since all the BackendN will pull from the same queue, they'll each have ~1/4 the stats that the combined one does now, and would only compare to the receive-side stats in the aggregate.
Change 586111 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] update vhtcpd exporter to match reality
Change 586111 merged by CDanis:
[operations/puppet@production] update vhtcpd exporter to match reality