Page MenuHomePhabricator

Upgrade varnish automatically via puppet in Beta Cluster
Closed, ResolvedPublic

Description

From email discussion with Brandon Black re bug 73357:

Brandon said:

Greg said:

Can we make sure that when we upgrade Varnish on prod we do so on Beta
Cluster as well? Maybe even doing it first on Beta Cluster to test
things :) But I'll leave specifics to you.

Yes! :) Perhaps ideally, we should have puppet auto-upgrading to the latest
package revs in beta. The only reason we don't do that in prod is because
there are outage impacts from restarts, so the upgrade->restart cycle has to be
paced out manually after testing. But for now, I'll try to remember to include
beta.


Version: unspecified
Severity: enhancement

Details

Reference
bz73564

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:51 AM
bzimport set Reference to bz73564.

Mukunda: Could you take a stab at this one with feedback from Brandon and Antoine?

CCing Brandon

The straight forward way would be to have puppet auto-upgrade varnish on beta cluster. Maybe using hiera() to define ensure => latest only on labs.

There is a couple oddities if I remember properly:

  • the package upgrade override some of our puppet provided files (init defaults, some config). So it needs to be restarted again once puppet reapplied the conf
  • if the new version of vagrant is not back compatible with our conf, we need the conf to be merged first.

Take them with a grain of salt though.

I would love release and ops team to write down the way Vagrant related changes are pushed to production. From there come up with a nicer strategy that might involve beta cluster and some integration tests as a stage before the change land to production.

Ok I've added version: latest to hiera on wikitech.I'll look into what it will take to deal with restarting after conf changes as @hashar suggested above. Maybe this is all that's really needed?

^ typo, should be Hiera: not Heira: :)

^ typo, should be Hiera: not Heira: :)

Thanks! typo is now fixed, good catch!

Well it sounds like you fixed this in hiera? In any case, the current problem with varnish versions in beta is this: T98758

All beta cluster varnish caches have been upgraded to Jessie (T98758 and T103660).

The Debian unattended-upgrades system is apparently enabled, so potentially new versions of varnish would be magically upgraded.

hashar claimed this task.

The last version is varnish (3.0.6plus-wm8) from 23 Sep 2015 so unattended upgrade hasn't been triggered for that specific package.

deployment-cache-text04 shows some activity under /var/log/unattended-upgrades/ so it should be running fine.