Page MenuHomePhabricator
Create Task
Maniphest T263557

Upgrade production cache nodes to Varnish 6
Closed, ResolvedPublic
Actions

Description

With T260702 and T261632 out of the way, after a brief diversion due to T261487, we can now start upgrading production nodes from 5.1.3-1wm15 to 6.0.6-1wm1.

Preliminary testing on traffic-cache-atstext-buster.traffic.eqiad.wmflabs is encouraging. With this patch cherry-picked (to be merged soon) I've upgraded varnish using the following procedure:

# Set profile::cache::varnish::frontend::packages_component: component/varnish6
run-puppet-agent
apt -y install libvmod-netmapper libvmod-re2 varnish-dbg varnish-modules varnish varnishkafka
systemctl stop varnish-frontend
run-puppet-agent

In case of necessity we can of course downgrade back from v6 to v5:

# Unset profile::cache::varnish::frontend::packages_component
rm /etc/apt/sources.list.d/repository_varnish.list
apt update
apt remove varnish libvmod-re2 -y
systemctl stop varnish-frontend
run-puppet-agent

Details

SubjectRepoBranchLines +/-
cache: upgrade Varnish to v6 in esamsoperations/puppetproduction+1 -10
cache: upgrade Varnish to v6 in codfwoperations/puppetproduction+2 -0
cache: upgrade Varnish to v6 in eqiadoperations/puppetproduction+2 -0
cache: upgrade Varnish to v6 in eqsinoperations/puppetproduction+2 -0
cache: upgrade Varnish to v6 in ulsfooperations/puppetproduction+0 -1
cp4021: upgrade to Varnish 6 (cache_upload)operations/puppetproduction+1 -0
varnish: upgrade upload labs node to v6operations/puppetproduction+1 -0
varnish: use component/varnish6 for VTC testsoperations/puppetproduction+1 -0
cp4027: upgrade to Varnish 6operations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.Sep 22 2020, 3:38 PM
Restricted Application added a project: SRE. · View Herald TranscriptSep 22 2020, 3:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Sep 22 2020, 3:38 PM
• ema moved this task from Backlog to Caching on the Traffic board.Sep 23 2020, 9:00 AM
• ema added a comment.Sep 23 2020, 10:01 AM

I've upgraded deployment-cache-text06 to Varnish 6, and https://en.wikipedia.beta.wmflabs.org looks fine. Later today I'll use Varnish 6 to run our VTC tests, and then proceed with the upgrade of a production node.

gerritbot added a comment.Sep 23 2020, 11:55 AM

Change 629366 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp4027: upgrade to Varnish 6

https://gerrit.wikimedia.org/r/629366

gerritbot added a project: Patch-For-Review.Sep 23 2020, 11:55 AM
gerritbot added a comment.Sep 23 2020, 11:58 AM

Change 629368 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: use component/varnish6 for VTC tests

https://gerrit.wikimedia.org/r/629368

gerritbot added a comment.Sep 23 2020, 12:06 PM

Change 629366 merged by Ema:
[operations/puppet@production] cp4027: upgrade to Varnish 6

https://gerrit.wikimedia.org/r/629366

gerritbot added a comment.Sep 23 2020, 12:06 PM

Change 629368 merged by Ema:
[operations/puppet@production] varnish: use component/varnish6 for VTC tests

https://gerrit.wikimedia.org/r/629368

• ema added a comment.Sep 23 2020, 12:08 PM

All cache_text VTC tests green with 6.0.6, proceeding with the upgrade of cp4027.

Stashbot added a comment.Sep 23 2020, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-23T12:09:14Z] <ema> cp4027: depool and upgrade varnish to 6.0.6-1wm1 T263557

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2020, 12:10 PM
Stashbot added a comment.Sep 23 2020, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-23T12:22:46Z] <ema> cp4027: repool with varnish 6.0.6-1wm1 T263557

• ema renamed this task from Upgrade a production cache node to Varnish 6 to Upgrade production cache nodes to Varnish 6.Sep 24 2020, 8:26 AM
• ema updated the task description. (Show Details)
• ema added a subscriber: • Gilles.Sep 24 2020, 8:29 AM

cp4027 has been running fine since yesterday with Varnish 6.0.6. Performance-wise there's no impact either, I've added a panel with p75 response time comparison to response-time-by-host (thanks @Gilles for fixing the data source!):

Screenshot from 2020-09-24 10-26-32.png (1×2 px, 249 KB)

Continuing with the upgrades today.

gerritbot added a comment.Sep 24 2020, 8:39 AM

Change 629623 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: upgrade upload labs node to v6

https://gerrit.wikimedia.org/r/629623

gerritbot added a project: Patch-For-Review.Sep 24 2020, 8:39 AM

Change 629623 merged by Ema:
[operations/puppet@production] varnish: upgrade upload labs node to v6

https://gerrit.wikimedia.org/r/629623

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2020, 9:10 AM
gerritbot added a comment.Sep 24 2020, 9:12 AM

Change 629634 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp4021: upgrade to Varnish 6 (cache_upload)

https://gerrit.wikimedia.org/r/629634

gerritbot added a project: Patch-For-Review.Sep 24 2020, 9:12 AM

Change 629634 merged by Ema:
[operations/puppet@production] cp4021: upgrade to Varnish 6 (cache_upload)

https://gerrit.wikimedia.org/r/629634

Stashbot added a comment.Sep 24 2020, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:14:22Z] <ema> cp4021: depool and upgrade varnish to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 24 2020, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:19:41Z] <ema> cp4021: redepool with varnish to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 24 2020, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T09:20:42Z] <ema> cp4021: repool with varnish 6.0.6-1wm1 T263557

gerritbot added a comment.Sep 24 2020, 10:06 AM

Change 629642 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in ulsfo

https://gerrit.wikimedia.org/r/629642

gerritbot added a comment.Sep 24 2020, 11:57 AM

Change 629642 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in ulsfo

https://gerrit.wikimedia.org/r/629642

Stashbot added a comment.Sep 24 2020, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:02:22Z] <ema> cp4022: upgrade varnish to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 24 2020, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:09:10Z] <ema> text@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2020, 12:10 PM
Stashbot added a comment.Sep 24 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-24T12:30:39Z] <ema> upload@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557

• ema added a comment.Sep 24 2020, 1:46 PM

ulsfo upgraded!

gerritbot added a comment.Sep 25 2020, 8:42 AM

Change 630083 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in eqsin

https://gerrit.wikimedia.org/r/630083

gerritbot added a project: Patch-For-Review.Sep 25 2020, 8:42 AM
gerritbot added a comment.Sep 25 2020, 8:50 AM

Change 630083 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in eqsin

https://gerrit.wikimedia.org/r/630083

Stashbot added a comment.Sep 25 2020, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-25T09:02:03Z] <ema> text@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 25 2020, 9:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-25T09:22:20Z] <ema> upload@eqsin: rolling varnish upgrade to 6.0.6-1wm1 T263557

• ema mentioned this in T236754: Discarded VCL files stuck in auto/busy state cause high number of backend probe requests.Sep 25 2020, 1:28 PM
Stashbot added a comment.Sep 28 2020, 8:21 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T08:21:31Z] <ema> upload@eqiad: rolling varnish upgrade to 6.0.6-1wm1 T263557

gerritbot added a comment.Sep 28 2020, 8:23 AM

Change 630540 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in eqiad

https://gerrit.wikimedia.org/r/630540

gerritbot added a comment.Sep 28 2020, 8:26 AM

Change 630540 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in eqiad

https://gerrit.wikimedia.org/r/630540

Stashbot added a comment.Sep 28 2020, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T08:32:00Z] <ema> text@eqiad: rolling varnish upgrade to 6.0.6-1wm1 T263557

gerritbot added a comment.Sep 28 2020, 9:25 AM

Change 630550 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in codfw

https://gerrit.wikimedia.org/r/630550

gerritbot added a comment.Sep 28 2020, 9:29 AM

Change 630550 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in codfw

https://gerrit.wikimedia.org/r/630550

Stashbot added a comment.Sep 28 2020, 9:29 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T09:29:52Z] <ema> text@codfw: rolling varnish upgrade to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 28 2020, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T09:48:30Z] <ema> upload@codfw: rolling varnish upgrade to 6.0.6-1wm1 T263557

gerritbot added a comment.Sep 28 2020, 11:26 AM

Change 630566 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: upgrade Varnish to v6 in esams

https://gerrit.wikimedia.org/r/630566

gerritbot added a comment.Sep 29 2020, 7:46 AM

Change 630566 merged by Ema:
[operations/puppet@production] cache: upgrade Varnish to v6 in esams

https://gerrit.wikimedia.org/r/630566

Stashbot added a comment.Sep 29 2020, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-29T08:01:35Z] <ema> cp3050: varnish upgrade to 6.0.6-1wm1 T263557

• ema mentioned this in T264074: varnishkafka 1.1.0 CPU usage increase.Sep 29 2020, 8:57 AM
Stashbot added a comment.Sep 29 2020, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-29T13:36:11Z] <ema> upload@esams: rolling varnish upgrade to 6.0.6-1wm1 T263557

Stashbot added a comment.Sep 29 2020, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-29T13:47:10Z] <ema> text@esams: rolling varnish upgrade to 6.0.6-1wm1 T263557

• ema closed this task as Resolved.Oct 1 2020, 7:26 AM
• ema claimed this task.

All production nodes are now running Varnish 6.0.6-1wm1. Closing!

• Gilles mentioned this in T264398: 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1).Oct 2 2020, 8:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits