Page MenuHomePhabricator

Upgrade production cache nodes to Varnish 6
Closed, ResolvedPublic

Description

With T260702 and T261632 out of the way, after a brief diversion due to T261487, we can now start upgrading production nodes from 5.1.3-1wm15 to 6.0.6-1wm1.

Preliminary testing on traffic-cache-atstext-buster.traffic.eqiad.wmflabs is encouraging. With this patch cherry-picked (to be merged soon) I've upgraded varnish using the following procedure:

# Set profile::cache::varnish::frontend::packages_component: component/varnish6
run-puppet-agent
apt -y install libvmod-netmapper libvmod-re2 varnish-dbg varnish-modules varnish varnishkafka
systemctl stop varnish-frontend
run-puppet-agent

In case of necessity we can of course downgrade back from v6 to v5:

# Unset profile::cache::varnish::frontend::packages_component
rm /etc/apt/sources.list.d/repository_varnish.list
apt update
apt remove varnish libvmod-re2 -y
systemctl stop varnish-frontend
run-puppet-agent

Event Timeline

ema created this task.Sep 22 2020, 3:38 PM
Restricted Application added a project: Operations. · View Herald TranscriptSep 22 2020, 3:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Sep 22 2020, 3:38 PM
ema moved this task from Triage to Caching on the Traffic board.Sep 23 2020, 9:00 AM
ema added a comment.Sep 23 2020, 10:01 AM

I've upgraded deployment-cache-text06 to Varnish 6, and https://en.wikipedia.beta.wmflabs.org looks fine. Later today I'll use Varnish 6 to run our VTC tests, and then proceed with the upgrade of a production node.

Change 629366 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp4027: upgrade to Varnish 6

https://gerrit.wikimedia.org/r/629366

Change 629368 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: use component/varnish6 for VTC tests

https://gerrit.wikimedia.org/r/629368

Change 629366 merged by Ema:
[operations/puppet@production] cp4027: upgrade to Varnish 6

https://gerrit.wikimedia.org/r/629366

Change 629368 merged by Ema:
[operations/puppet@production] varnish: use component/varnish6 for VTC tests

https://gerrit.wikimedia.org/r/629368

ema added a comment.Sep 23 2020, 12:08 PM

All cache_text VTC tests green with 6.0.6, proceeding with the upgrade of cp4027.

Mentioned in SAL (#wikimedia-operations) [2020-09-23T12:09:14Z] <ema> cp4027: depool and upgrade varnish to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-23T12:22:46Z] <ema> cp4027: repool with varnish 6.0.6-1wm1 T263557

ema renamed this task from Upgrade a production cache node to Varnish 6 to Upgrade production cache nodes to Varnish 6.Sep 24 2020, 8:26 AM
ema updated the task description. (Show Details)
ema added a subscriber: Gilles.Sep 24 2020, 8:29 AM

cp4027 has been running fine since yesterday with Varnish 6.0.6. Performance-wise there's no impact either, I've added a panel with p75 response time comparison to response-time-by-host (thanks @Gilles for fixing the data source!):

Continuing with the upgrades today.

Change 629623 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: upgrade upload labs node to v6

https://gerrit.wikimedia.org/r/629623

Change 629623 merged by Ema:
[operations/puppet@production] varnish: upgrade upload labs node to v6

https://gerrit.wikimedia.org/r/629623

Change 629634 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp4021: upgrade to Varnish 6 (cache_upload)

https://gerrit.wikimedia.org/r/629634

Change 629634 merged by Ema:
[operations/puppet@production] cp4021: upgrade to Varnish 6 (cache_upload)

https://gerrit.wikimedia.org/r/629634

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:14:22Z] <ema> cp4021: depool and upgrade varnish to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:19:41Z] <ema> cp4021: redepool with varnish to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:20:42Z] <ema> cp4021: repool with varnish 6.0.6-1wm1 T263557

Change 629642 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in ulsfo

https://gerrit.wikimedia.org/r/629642

Change 629642 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in ulsfo

https://gerrit.wikimedia.org/r/629642

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:02:22Z] <ema> cp4022: upgrade varnish to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:09:10Z] <ema> text@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:30:39Z] <ema> upload@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557

ema added a comment.Sep 24 2020, 1:46 PM

ulsfo upgraded!

Change 630083 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in eqsin

https://gerrit.wikimedia.org/r/630083

Change 630083 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in eqsin

https://gerrit.wikimedia.org/r/630083

Mentioned in SAL (#wikimedia-operations) [2020-09-25T09:02:03Z] <ema> text@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-25T09:22:20Z] <ema> upload@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-28T08:21:31Z] <ema> upload@eqiad: rolling varnish upgrade to 6.0.6-1wm1 T263557

Change 630540 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in eqiad

https://gerrit.wikimedia.org/r/630540

Change 630540 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in eqiad

https://gerrit.wikimedia.org/r/630540

Mentioned in SAL (#wikimedia-operations) [2020-09-28T08:32:00Z] <ema> text@eqiad: rolling varnish upgrade to 6.0.6-1wm1 T263557

Change 630550 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in codfw

https://gerrit.wikimedia.org/r/630550

Change 630550 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in codfw

https://gerrit.wikimedia.org/r/630550

Mentioned in SAL (#wikimedia-operations) [2020-09-28T09:29:52Z] <ema> text@codfw: rolling varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-28T09:48:30Z] <ema> upload@codfw: rolling varnish upgrade to 6.0.6-1wm1 T263557

Change 630566 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in esams

https://gerrit.wikimedia.org/r/630566

Change 630566 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in esams

https://gerrit.wikimedia.org/r/630566

Mentioned in SAL (#wikimedia-operations) [2020-09-29T08:01:35Z] <ema> cp3050: varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-29T13:36:11Z] <ema> upload@esams: rolling varnish upgrade to 6.0.6-1wm1 T263557

Mentioned in SAL (#wikimedia-operations) [2020-09-29T13:47:10Z] <ema> text@esams: rolling varnish upgrade to 6.0.6-1wm1 T263557

ema closed this task as Resolved.Thu, Oct 1, 7:26 AM
ema claimed this task.

All production nodes are now running Varnish 6.0.6-1wm1. Closing!