The "Check Varnish expiry mailbox lag" Icinga alert on cp4021 is in status UNKNOWN because:
"UNKNOWN: cannot run varnishstat"
This started about 10 hours ago from creating this ticket.
Manually running it:
root@cp4021:~# varnishstat Can't open log - retrying for 5 seconds Cannot open /var/lib/varnish/cp4021/_.vsm: No such file or directory
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| cache: move check_varnish_expiry_mailbox_lag to backend profile | operations/puppet | production | +19 -19 |
that's expected, as @ema mentioned yesterday in -traffic:
<ema> so we've got cp4021 reimaged as Varnish/ATS and it seems to be looking kind-of OK <ema> it is however still depooled as I haven't had the chance to look deeply at everything, and surely certain things are missing (like prometheus metrics not showing up in grafana yet) also, we assume that all nodes in cache::upload::nodes need to both be listed as backends for varnish-fe and be involved in ipsec shenanigans the latter isn't true anymore with ATS, so that needs to be fixed too <ema> (I've just ack'ed the alerts for now) <ema> but we're getting close! :)
Indeed our Varnish mailbox lag Icinga check only applies to Varnish backends, given that backends are those affected by T145661 and similar issues. During the Puppet refactoring splitting frontend/backend puppetization (T219967) I forgot to move the check from the Varnish module, where it shouldn't have been in the first place, to the backend profile. Doing this will ensure that the check is only added to cache hosts using Varnish as the cache backend software, not those using ATS such as cp4021.
Change 506090 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move check_varnish_expiry_mailbox_lag to backend profile
Change 506090 merged by Ema:
[operations/puppet@production] cache: move check_varnish_expiry_mailbox_lag to backend profile