Unfortunately, as discussed multiple times in the past, reprepro does not really allow us to easily have more than one version of a package without some tricks.

Production pins the version in the puppet config to avoid having unscheduled upgrades indeed. The pin is done in hiera, see https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/hieradata/role/common/kubernetes/worker.yaml. It will probably soon become "present" though, decoupling puppet from the installed version.

which of course does not work very well when a new version is uploaded on our repo. Puppet fails for that resource then as it is not able to find the pinned version. We could move the pinning to the apt repo updating. An example would be rOPUP848d9a9badd6d58cde7eed0c21363b83c828406c. This would have the following consequences

Pros:

• It would decouple the installed version from the available version in apt.
• Puppet would not complain when the version in the repo changes since it would be in "present" mode (there is no point in having pin in both places).
• Scheduled upgrades would happen way easier and orchestrated since puppet would no longer force the version.
• Labs and production would have to have the same version of the software for prolonged periods of time, but with the capability of short periods of time diverging from that.

Cons:

• There could be unwanted upgrades when the version changes and somebody does a apt-get upgrade in the whole box

@yuvipanda does this sound acceptable ?

@akosiaris hmm I'd really like to keep the pin in puppet - there's enough uncertainity as is without having to find docker version mismatches. Upgrades will also restart all containers, and making 'apt-get upgrade' be that destructive seems like a pretty big negative to me.

If we have only one version it also means we are tying the prod and tools versions together forever, with upgrades needing to happen at the same tie.

But reprepro somewhat supports multiple versions as long as they're stored in different sections (or whatever the exact terminology is for reprepro, I'm referring to backports/main/thirdparty/experimental). When I imported nodejs 6 I wanted to still be able to make nodejs 4 available for kartotherian (which has tech debt which prevents it from going to node 6 now); nodejs 4 was previously imported in "backports", so I imported nodejs 6 into "main". Both versions are available on a jessie system and can be selectively installed via "apt-get install nodejs=VERSION". Maybe I'm missing something here, but should work for docker as well?

We'll have to create maybe a 'labs' section in reprepo and use it?

That or maybe "staging" to make it a little more generic.

Yeah, I thought about the section workaround too, but at best it's a hack. And tbh, I am not in love with the idea of a "labs" section at all. Getting a "staging" section sounds way better, so it might be ok. Adding @faidon since he has defined the various sections we currently have.

In T153416#2955290, @yuvipanda wrote:

@akosiaris hmm I'd really like to keep the pin in puppet - there's enough uncertainity as is without having to find docker version mismatches. Upgrades will also restart all containers, and making 'apt-get upgrade' be that destructive seems like a pretty big negative to me.

Just noting that without the pinning in reprepro as well, the puppet pin will just rollback the package version. So without a repo pin, apt-get upgrade remains destructive. Which is why I am not in love with the version pinning in puppet.

On further thought, I think I just want to use the aptly that we've setup for tools already.

1. We already use this for other packages we need, so this will keep it all in one place
2. There are tools volunteer admins who have access to aptly and not reprepro
3. I don't know / think anyone in the Tools admin team has much experience with Reprepo, but we do with Aptly.

Is there really much experience with aptly? :-) I stumble along with it quite a bit, and – if possible – I would much rather switch the aptly repository to reprepro, or even move the Toolforge packages into the WMF repository so that (in the Labs ecosphere) there are more eyes familiar with the procedures.

IMHO updates of Docker & Co. should not be done without some staffer in attendance (and probably scheduled well in advance), so the access question does not seem relevant to me.

I just manually imported docker-engine version tools is using into aptly, and it all seems to be ok.

Do we have a verdict on this one ?

Some higher level plans for structuring the repos are now collected at https://phabricator.wikimedia.org/T158583, input welcome.

There's a thread on ops-l now because docker is now docker community
edition.