Although we did briefly discuss the results of this experiment within Traffic, I don't think we ever publicly disclosed our analysis.

We should probably add a Conflicts: libvarnishapi1 to our varnish 6 packaging, or whatever relationship magic is the right one to ensure that if varnish 6 is installed, libvarnishapi1 is not.

Apparently deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud is gone and has been replaced by deployment-mediawiki12.

Smoke testing of 6.0.9 is fine on deployment-prep, I'll start upgrading production nodes next week.

In T298758#7604333, @gerritbot wrote:

Change 752151 had a related patch set uploaded (by Ema; author: Ema):

[operations/debs/varnish4@debian-wmf] Use libunwind for backtraces

https://gerrit.wikimedia.org/r/752151

Hello @Ade56facc and @Yann, thanks for the bug reports.

Many of the assumptions made when this task was created have changed since the migration to ATS for cache backends (no more IPSec, the difference between Tier1 and Tier2 DCs is now gone, ...). We are now in a world where all backend caches access the origins via TLS, which I think largely covers what we wanted to achieve here. @BBlack: I'm marking the task as resolved, but of course feel free to reopen / create other tasks as needed if you think that anything is missing.

In the last 24 hours we had just one overrun on 4 nodes:

The situation has improved significantly, we are now processing up to 13K lines per second vs the ~8K plateau from last week:

After setting cache::single_backend_fqdn: cp4021.ulsfo.wmnet in hiera, cp4021 is now gone from the list of cache backends on all upload@ulsfo nodes, see for instance cp4022:

This is a recurring problem, see for example T273599 T222072 T295253. T222075 has ideas on how to tackle the issue. I've tried to access the instance to free up some space but I don't seem to have the permissions to do so.

In T295120#7484351, @MoritzMuehlenhoff wrote:

The puppet code lacks a "priority => 1002", if you want to override "main" (which also has priority=1001). See the comments in the apt::package_from_component define for further context.

The optimizations to varnishxcache.mtail and varnishreqstats.mtail paid off, time spent in tryBacktrack has decreased significantly:

In T293879#7460663, @gerritbot wrote:

Change 734893 merged by Ema:

[operations/puppet@production] varnishxcache.mtail: avoid unnecessary filtering

https://gerrit.wikimedia.org/r/734893

In T293879#7454657, @gerritbot wrote:

Change 732925 merged by Ema:

[operations/puppet@production] varnishttfb.mtail: use native histogram type

https://gerrit.wikimedia.org/r/732925

I upgraded varnish to 6.0.8 everywhere (see T292290) and forgot about restarting the service on deployment-cache-upload06. It should be fixed now, thanks @Majavah.

In T293879#7450109, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2021-10-22T08:23:48Z] <ema> cp3062: test 0008-vsl_check_e_inval_assertion.patch https://gerrit.wikimedia.org/r/c/operations/debs/varnish4/+/732913/ T293879

I've tried using a separate mtail instance with a subset of the scripts used by the production instance, namely:

By giving a very large amount - 3G instead of the default 80M - of vsl_space to cp3062, the issue happens less often but still does happen. On all text@esams nodes, there has been no overrun between 00:44 and 04:46 (when esams traffic is at its lowest), while on cp3062 the last overrun happened at 21:56 and the first one this EU morning at 06:38. Bumping vsl_space alone does not fix the issue.

Trying the lowest possible hanging fruit first, namely rising vsl_space. I've first tried setting it to 512M as mentioned in the SAL entry above, but that failed due to the size of /var/lib/varnish (512M, but it needs space for other stuff too). We now have vsl_space=480M on cp3062, let's see if that changes anything at all.

All hosts upgraded.

Caches have now filled up. Response start looks good on cp3060 compared to one week ago:

Setting priority to low for now as these seem isolated, sporadic crashes and systemd took care of the restarts as expected so there was no production impact.

In T292290#7418496, @Krinkle wrote:

I've made some improvements to the by-host dash that may be of use:
https://grafana.wikimedia.org/d/M7xQ_BeWk/response-time-by-host

Trying another reimage as follows:

Runbook and updated dashboard link are shown correctly. Closing.

@bd808: looks like we're all set!

Runbook created: https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop

Heads up Performance-Team: as with all Varnish upgrades, this may have an impact (positive or negative) on performance. You may want to keep the upgrade process on your radar: beta has been running with Varnish 6.0.8 for a few days now without obvious issues, I'll upgrade one prod text node in ulsfo today and then carry on relatively quickly unless things break.

In T288106#7405557, @gerritbot wrote:

Change 726912 had a related patch set uploaded (by Ema; author: Ema):

[operations/puppet@production] cache: exclude single backend experiment from pooled ATS backends

https://gerrit.wikimedia.org/r/726912

After lowering the amount of memory used for the ATS backend ram cache, there's now some more available on the system:

Preliminary testing in beta looks good, uploading the package to the archive.