Page MenuHomePhabricator

Package and deploy Varnish 6.0.8
Closed, ResolvedPublic

Description

The latest Varnish LTS release is 6.0.8. The new version includes the following changes:

  • Fix an issue where a backend fetch can stall after a client has disconnected. (3556)
  • Fix an issue in directors.shard() where calling reconfigure() on an empty director breaks subsequent shard directors. Also changed an internal canon_point calculation. (3593)
  • Address and fix various issues with H2 work priorities. (2796, 3536, 3537)
  • Fix a panic situation in ban_mark_completed() when using bans. (2556)
  • Fix an issue where an early backend error can trigger a valid zero length response. (3560)
  • Fix a panic situation when cooling a backend in VBE_Poll() and Lck_Delete(). (3587)
  • Complete source code migration to python3.
  • Fix an issue where varnishd will sleep for 1s when attempting to shutdown. (3569)
  • Fix a panic situation in vbf_stp_condfetch(). (3558)
  • Fix a panic situation in EXP_Insert(). (2999)
  • Fix a panic situation in VRB_Free() which can be triggered when using std.cache_req_body(). (3433)
  • Fix a panic situation in http1_minimal_response(). (3415)
  • Fix an issue where a closed connection gets recycled for reuse. (3400)
  • Fix an issue where directors.round_robin() can sometimes be empty when a single backend is sick. (3474)
  • Fix an issue where the wrong Content-Length is used when doing gunzip on delivery. (3535)
  • Fix an issue where resp.reason can race when used in vcl_synth. (3546)
  • Fix an issue where the return(error) status and reason are kept when doing a backend retry. (3525)
  • Fix an HTTP/2.0 request smuggling vulnerability. (VSV00007)

This task tracks the packaging and deployment work required to upgrade to 6.0.8.

Event Timeline

ema triaged this task as Medium priority.Fri, Oct 1, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-05T12:24:17Z] <ema> deployment-cache-text06: upgrade varnish to 6.0.8-1wm1 T292290

Preliminary testing in beta looks good, uploading the package to the archive.

Mentioned in SAL (#wikimedia-operations) [2021-10-05T12:53:34Z] <ema> upload varnish 6.0.8-1wm1 to apt.wikimedia.org T292290

Heads up Performance-Team: as with all Varnish upgrades, this may have an impact (positive or negative) on performance. You may want to keep the upgrade process on your radar: beta has been running with Varnish 6.0.8 for a few days now without obvious issues, I'll upgrade one prod text node in ulsfo today and then carry on relatively quickly unless things break.

Mentioned in SAL (#wikimedia-operations) [2021-10-11T12:45:16Z] <ema> cp4027: upgrade varnish to 6.0.8 T292290

I've made some improvements to the by-host dash that may be of use:
https://grafana.wikimedia.org/d/M7xQ_BeWk/response-time-by-host

I've made some improvements to the by-host dash that may be of use:
https://grafana.wikimedia.org/d/M7xQ_BeWk/response-time-by-host

Thanks!

The cache on cp4027 has filled up and things look good. Upgrading one text and one upload node in esams now to observe the new release on a busy DC.

Mentioned in SAL (#wikimedia-operations) [2021-10-18T07:34:07Z] <ema> cp3060 (text), cp3061 (upload): upgrade varnish to 6.0.8 T292290

Caches have now filled up. Response start looks good on cp3060 compared to one week ago:

cp3060-cp3062-respStart.png (819×1 px, 124 KB)

In general all interesting metrics look reasonable on cache-hosts-comparison too. Starting with the rolling restarts today.

Mentioned in SAL (#wikimedia-operations) [2021-10-19T07:24:27Z] <ema> A:cp start rolling varnish upgrades to 6.0.8-1wm1 T292290

ema claimed this task.

All hosts upgraded.