Page MenuHomePhabricator

Package varnish 6.0.x
Closed, ResolvedPublic

Description

T260702 describes the work related to our custom varnish 5 patches that can/cannot be dropped, moving to varnish 6. More in general, we need to rebuild/update the following source packages:

libvmod-tbf this one is no longer in use

Event Timeline

ema created this task.Aug 31 2020, 11:47 AM
Restricted Application added a project: SRE. · View Herald TranscriptAug 31 2020, 11:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema added a comment.Aug 31 2020, 12:56 PM

When it comes to varnish-modules, our current version (0.12.1-1+wmf2) does not build against 6.0.x, and same goes for varnish-modules 0.16.0 currently in testing. Luckily though, with a few changes 0.15.0-1 from buster does work, see P12422.

CDanis added a subscriber: CDanis.Aug 31 2020, 12:57 PM

Change 623396 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-tbf@debian] Release 2.0.91-3wm

https://gerrit.wikimedia.org/r/623396

ema moved this task from Triage to Caching on the Traffic board.Sep 1 2020, 8:27 AM
ema added a comment.Sep 1 2020, 8:36 AM

Varnish 6.0.x fails to start in labs with our current setup:

Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.Arg.8ccd32f49707a395': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.Arg.49ef770c0aaa3bbb': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.Stat.f8ce5151613a4db7': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.StatDoc.af11865a84dc74bd': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.Arg.a0cb26490e44118c': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: rm: cannot remove '_.vsm_mgt/_.index': Permission denied
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]: Assert error in mgt_SHM_Init(), mgt/mgt_shmem.c line 96:
Sep 01 07:42:53 traffic-cache-atstext-buster varnish-frontend[7196]:   Condition((system("rm -rf " "_.vsm_mgt")) == 0) not true.

This is due to our current systemd settings for CapabilityBoundingSet:

CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_NET_BIND_SERVICE CAP_KILL

We need to allow root to skip read/write/execute permission checks by giving it back CAP_DAC_OVERRIDE, as the directory is owned by the varnish user:

drwxr-x--- 2 varnish varnish 160 Sep  1 08:24 /var/lib/varnish/frontend/_.vsm_mgt/

Change 623531 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: give CAP_DAC_OVERRIDE back to root

https://gerrit.wikimedia.org/r/623531

Change 623533 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: do not explicitly install libvarnishapi1

https://gerrit.wikimedia.org/r/623533

Change 623583 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: stop installing libvmod-tbf

https://gerrit.wikimedia.org/r/623583

Vgutierrez updated the task description. (Show Details)Sep 1 2020, 1:15 PM

Change 623614 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-re2@debian] Release 1.3.1-4

https://gerrit.wikimedia.org/r/623614

Change 623531 merged by Ema:
[operations/puppet@production] varnish: give CAP_DAC_OVERRIDE back to root

https://gerrit.wikimedia.org/r/623531

Change 623533 merged by Ema:
[operations/puppet@production] varnish: do not explicitly install libvarnishapi1

https://gerrit.wikimedia.org/r/623533

Change 623583 merged by Ema:
[operations/puppet@production] varnish: stop installing libvmod-tbf

https://gerrit.wikimedia.org/r/623583

Change 625629 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-re2@debian-6.0] Release 1.5.3-1

https://gerrit.wikimedia.org/r/625629

Change 623614 abandoned by Vgutierrez:
[operations/software/varnish/libvmod-re2@debian] Release 1.3.1-4

Reason:
we need to ship libvmod-re2 1.5.3

https://gerrit.wikimedia.org/r/623614

Vgutierrez updated the task description. (Show Details)Sep 7 2020, 3:04 PM

Change 625659 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-netmapper@debian] 1.7-4: Rebuild against Varnish 6

https://gerrit.wikimedia.org/r/625659

Change 625713 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-netmapper@master] Fix includes to build against Varnish 6

https://gerrit.wikimedia.org/r/625713

Change 625843 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/libvmod-netmapper@master] 1.8: Bump version number

https://gerrit.wikimedia.org/r/625843

Change 625713 merged by Vgutierrez:
[operations/software/varnish/libvmod-netmapper@master] Fix includes to build against Varnish 6

https://gerrit.wikimedia.org/r/625713

Change 625843 merged by Vgutierrez:
[operations/software/varnish/libvmod-netmapper@master] 1.8: Bump version number

https://gerrit.wikimedia.org/r/625843

Vgutierrez updated the task description. (Show Details)Sep 8 2020, 1:58 PM

Change 626177 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/varnish/varnishkafka@debian] varnishkafka 1.0.15

https://gerrit.wikimedia.org/r/626177

Vgutierrez updated the task description. (Show Details)Sep 10 2020, 2:39 PM

@ema I've added to the task description the CRs required to get the packages of all the vmods and varnishkafka, I've seen that we have varnish-modules compiled on deneb but I haven't found a CR.

that should be enough to test varnish6 on the labs environment:

vgutierrez@deneb:~$ ls /var/cache/pbuilder/result/buster-amd64/*varnish*.deb
/var/cache/pbuilder/result/buster-amd64/libvarnishapi-dev_6.0.6-1wm1_amd64.deb		/var/cache/pbuilder/result/buster-amd64/varnish-modules_0.12.1-1+wmf3_amd64.deb
/var/cache/pbuilder/result/buster-amd64/libvarnishapi2_6.0.6-1wm1_amd64.deb		/var/cache/pbuilder/result/buster-amd64/varnish-modules_0.15.0-1+wmf1_amd64.deb
/var/cache/pbuilder/result/buster-amd64/varnish-dbg_6.0.6-1wm1_amd64.deb		/var/cache/pbuilder/result/buster-amd64/varnish_6.0.6-1wm1_amd64.deb
/var/cache/pbuilder/result/buster-amd64/varnish-doc_6.0.6-1wm1_all.deb			/var/cache/pbuilder/result/buster-amd64/varnishkafka-dbg_1.0.15-1_amd64.deb
/var/cache/pbuilder/result/buster-amd64/varnish-modules-dbgsym_0.12.1-1+wmf3_amd64.deb	/var/cache/pbuilder/result/buster-amd64/varnishkafka_1.0.15-1_amd64.deb
/var/cache/pbuilder/result/buster-amd64/varnish-modules-dbgsym_0.15.0-1+wmf1_amd64.deb
vgutierrez@deneb:~$ ls /var/cache/pbuilder/result/buster-amd64/*vmod*.deb
/var/cache/pbuilder/result/buster-amd64/libvmod-netmapper-dbg_1.8-1_amd64.deb  /var/cache/pbuilder/result/buster-amd64/libvmod-re2-dbg_1.5.3-1_amd64.deb  /var/cache/pbuilder/result/buster-amd64/libvmod-tbf-dbg_2.0.91-3wm_amd64.deb
/var/cache/pbuilder/result/buster-amd64/libvmod-netmapper_1.8-1_amd64.deb      /var/cache/pbuilder/result/buster-amd64/libvmod-re2_1.3.1-3_amd64.deb	  /var/cache/pbuilder/result/buster-amd64/libvmod-tbf_2.0.91-3wm_amd64.deb
/var/cache/pbuilder/result/buster-amd64/libvmod-re2-dbg_1.3.1-3_amd64.deb      /var/cache/pbuilder/result/buster-amd64/libvmod-re2_1.5.3-1_amd64.deb

Change 629061 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] aptrepo: add component/varnish6

https://gerrit.wikimedia.org/r/629061

Change 629061 merged by Ema:
[operations/puppet@production] aptrepo: add component/varnish6

https://gerrit.wikimedia.org/r/629061

Mentioned in SAL (#wikimedia-operations) [2020-09-22T11:25:45Z] <ema> upload varnish 6.0.6-1wm1 to buster-wikimedia component/varnish6 T261632

Change 629094 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: install packages specifying archive component

https://gerrit.wikimedia.org/r/629094

Change 629103 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/varnish/varnishkafka@debian] 1.1.0-1: Varnish 6 support

https://gerrit.wikimedia.org/r/629103

Change 629104 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] package_builder: add hook D04varnish6

https://gerrit.wikimedia.org/r/629104

Change 629103 merged by Ema:
[operations/software/varnish/varnishkafka@debian] 1.1.0-1: Varnish 6 support

https://gerrit.wikimedia.org/r/629103

Mentioned in SAL (#wikimedia-operations) [2020-09-22T12:54:05Z] <ema> upload varnishkafka 1.1.0-1 to buster-wikimedia component/varnish6 T261632

Change 625659 merged by Ema:
[operations/software/varnish/libvmod-netmapper@debian] 1.8-1: Rebuild against Varnish 6

https://gerrit.wikimedia.org/r/625659

Mentioned in SAL (#wikimedia-operations) [2020-09-22T13:35:45Z] <ema> upload libvmod-netmapper 1.8-1 to buster-wikimedia component/varnish6 T261632

ema updated the task description. (Show Details)Sep 22 2020, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T13:55:08Z] <ema> upload varnish-modules 0.15.0-1+wmf1 to buster-wikimedia component/varnish6 T261632

Change 629104 merged by Ema:
[operations/puppet@production] package_builder: add hook D04varnish6

https://gerrit.wikimedia.org/r/629104

Change 625629 merged by Ema:
[operations/software/varnish/libvmod-re2@debian-6.0] Release 1.5.3-1

https://gerrit.wikimedia.org/r/625629

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:24:26Z] <ema> upload libvmod-re2 1.5.3-1 to buster-wikimedia component/varnish6 T261632

ema updated the task description. (Show Details)Sep 22 2020, 2:28 PM
ema closed this task as Resolved.Sep 22 2020, 3:27 PM
ema claimed this task.

All packages ready for prime time!

Change 629094 merged by Ema:
[operations/puppet@production] varnish: install packages specifying archive component

https://gerrit.wikimedia.org/r/629094

Change 626177 abandoned by Ema:
[operations/software/varnish/varnishkafka@debian] varnishkafka 1.0.15

Reason:
https://gerrit.wikimedia.org/r/c/operations/software/varnish/varnishkafka/ /629103

https://gerrit.wikimedia.org/r/626177