Change 275779 had a related patch set uploaded (by Ema):
Add basic support for varnishtest

https://gerrit.wikimedia.org/r/275779

Change 275779 merged by Ema:
Add basic support for varnishtest

https://gerrit.wikimedia.org/r/275779

Please note that there is no dependency on varnish 4 for this, we have already started adding VTC tests: https://gerrit.wikimedia.org/r/#/q/status:merged+project:operations/puppet+branch:production+topic:varnishtest,n,z

On a machine deployed with role cache::maps, tests can be executed as follows:

PATH=/bin:/usr/sbin:/usr/bin varnishtest /usr/share/varnish/tests/maps/*.vtc

@ema and @hashar going to pair on this and seek to get a proof of concept.

Change 276554 had a related patch set uploaded (by Hashar):
dib: provision varnish

https://gerrit.wikimedia.org/r/276554

Change 276554 merged by jenkins-bot:
dib: provision varnish

https://gerrit.wikimedia.org/r/276554

Mentioned in SAL [2016-03-10T20:19:34Z] <hashar> Refreshing Nodepool to include the 'varnish' package T128188

The Nodepool instances now have the varnish package at whatever version is defined by apt conf. which should be 3.x since it is 1001 pinned.

Notice: /Stage[main]/Main/Package[varnish]/ensure: ensure changed 'purged' to 'present'

Available since snapshot image ci-jessie-wikimedia-1457641052.

^^ that is just preliminary task to get /usr/bin/varnishtest available ;-}

As long as they're jessie machines that pull from the wikimedia package repos, it'll be the right version.

The Nodepool instances are indeed Jessie and reuse apt configuration from operations/puppet.git. I can confirm 3.x is installed on them:

varnish:
  Installed: 3.0.6plus-wm8
  Candidate: 3.0.6plus-wm8
  Version table:
     4.0.2-1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
 *** 3.0.6plus-wm8 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Now it is all about figuring out a command to ship in puppet.git and write a Jenkins job that invokes it ;-}

Change 278256 had a related patch set uploaded (by Ema):
Skip backends, not directors in test VCL code

https://gerrit.wikimedia.org/r/278256

Change 278256 merged by Ema:
Skip backends, not directors in test VCL code

https://gerrit.wikimedia.org/r/278256

Change 278948 had a related patch set uploaded (by Ema):
VTC tests compatible with Varnish 3 and 4

https://gerrit.wikimedia.org/r/278948

Change 278948 merged by Ema:
VTC tests compatible with Varnish 3 and 4

https://gerrit.wikimedia.org/r/278948

Change 279108 had a related patch set uploaded (by Ema):
Do not use dynamic directors in test VCL code

https://gerrit.wikimedia.org/r/279108

Change 279108 merged by Ema:
Do not use dynamic directors in test VCL code

https://gerrit.wikimedia.org/r/279108

Change 282137 had a related patch set uploaded (by Ema):
Misc cluster VTC tests

https://gerrit.wikimedia.org/r/282137

Change 282137 merged by Ema:
Misc cluster VTC tests

https://gerrit.wikimedia.org/r/282137

Change 282167 had a related patch set uploaded (by Ema):
Fix ownership of VTC tests directory

https://gerrit.wikimedia.org/r/282167

Change 282167 merged by Ema:
Fix ownership of VTC tests directory

https://gerrit.wikimedia.org/r/282167

Change 299543 had a related patch set uploaded (by Ema):
cache_upload VTC tests

https://gerrit.wikimedia.org/r/299543

Change 299543 merged by Ema:
cache_upload VTC tests

https://gerrit.wikimedia.org/r/299543

Change 506868 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: run VTC tests against remote PCC

https://gerrit.wikimedia.org/r/506868

Change 506868 merged by Ema:
[operations/puppet@production] varnish: run VTC tests against remote PCC

https://gerrit.wikimedia.org/r/506868

VTC tests can now be run from dev workstations against PCC:

ema@ariel:~/wmf/operations-puppet$ cd modules/varnish/files/tests ; ./run.py cp1078.eqiad.wmnet 506977
[*] running PCC for change 506977...
	PCC URL: https://puppet-compiler.wmflabs.org/compiler1002/16139/

[*] Dumping files...
	Catalog URL: https://puppet-compiler.wmflabs.org/compiler1002/16139//cp1078.eqiad.wmnet/change.cp1078.eqiad.wmnet.pson
	Creating etc/varnish/errorpage.inc.vcl
	Creating etc/varnish/browsersec.inc.vcl
	Creating etc/varnish/translation-engine.inc.vcl
	Creating etc/varnish/analytics.inc.vcl
	Creating etc/varnish/upload-common.inc.vcl
	Creating etc/varnish/normalize_path.inc.vcl
	Creating etc/varnish/wikimedia-common_upload-frontend.inc.vcl
	Creating etc/varnish/wikimedia_upload-frontend.vcl
	Creating usr/share/varnish/tests/wikimedia-common_upload-frontend.inc.vcl
	Creating usr/share/varnish/tests/wikimedia_upload-frontend.vcl
	Creating etc/varnish/upload-frontend.inc.vcl
	Creating usr/share/varnish/tests/upload-frontend.inc.vcl
	Creating etc/varnish/wikimedia-common_upload-backend.inc.vcl
	Creating etc/varnish/wikimedia_upload-backend.vcl
	Creating usr/share/varnish/tests/wikimedia-common_upload-backend.inc.vcl
	Creating usr/share/varnish/tests/wikimedia_upload-backend.vcl
	Creating etc/varnish/upload-backend.inc.vcl
	Creating usr/share/varnish/tests/upload-backend.inc.vcl

[*] Finding cluster...
	cp1078.eqiad.wmnet is a cache_upload host

[*] Running varnishtest (this might take a while)...
	sudo varnishtest -k -Dcc_command='exec gcc -std=gnu99 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -pthread -fpic -shared -Wl,-x -o %o %s -lmaxminddb' -Dbasepath=/home/ema/wmf/operations-puppet/modules/varnish/files/tests -Dvcl_path=/home/ema/wmf/operations-puppet/modules/varnish/files/tests/usr/share/varnish/tests:/home/ema/wmf/operations-puppet/modules/varnish/files/tests/etc/varnish upload/*.vtc

0 tests failed, 0 tests skipped, 15 tests passed
Test output saved to /tmp/tmp8tugl7nr

As mentioned in the README file, varnish and other packages must be installed from stretch-wikimedia for this to work.

@hashar can we run it in CI? The catalog needs to be compiled against a given hostname. Perhaps this could be passed with 'Host: cp1079.eqiad.wmnet' in the commit message?

In T128188#5143265, @ema wrote:

@hashar can we run it in CI? The catalog needs to be compiled against a given hostname. Perhaps this could be passed with 'Host: cp1079.eqiad.wmnet' in the commit message?

@hashar: bump! :)

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

The task is from 2016, and I most probably filed it as a placeholder to track effort to enhance test coverage on various repositories. We were on a rampage to add more tests everywhere at the time.

Ema and I talked a bit about it from time to time even though that is not necessarily reflected on this task. The challenge I think was to provide a way to setup the whole testing environment on CI, notably to expand Puppet templates. Running Puppet out of a Puppetmaster is not necessarily trivial since one needs to craft a bunch of parameters, setup Hiera lookup and we need the facts which are not publicly available.

Anyway. We do have facts on three CI instances which are used to run the Puppet Compiler: https://integration.wikimedia.org/ci/label/puppet-compiler-node/ so we can probably reuse those hosts and retrieve the facts from those hosts (how? I have no idea, they used to be flat YAML files but I think nowadays that is using a PuppetDB).

We would need a Docker image which has all the required Packages (Varnish? VCL Tester? Python). Probably crafted using docker-pkg and its definition stored in integration/config ./dockerfiles.

Some kind of entry point that CI will invoke ideally a rake task which will wrap up cd modules/varnish/files/tests ; ./run.py. This way the CI job is completely dumb and would just rake vcltest for example, delegating all the logic to developers in operations/puppet

Crafting the CI job would then boils down to docker run <theimage> and add it to trigger for operations/puppet.git (possibly restricted solely to changes altering modules/varnish/* files.