Page MenuHomePhabricator

Debian repository supporting multiple package versions
Closed, ResolvedPublic


Problem Statement #1

Once a new Debian package has been built it needs testing, at the moment this can be done for example by manually copying the package to labs and/or to canary hosts in production. While this works it is suboptimal, especially when dependencies are involved as well. A better option would be to have a separate distribution where to upload packages, this distribution would be installed only on particular hosts (e.g. canaries) and can be upgraded without affecting production. Once testing is complete the package can be copied to "$distribution-wikimedia" (in other words, promoted) using
see also

Problem Statement #2

Some clustered services (Cassandra for example), require identical software versions on constituent nodes. Having a new node provisioned with a newer version of software, simply because the repository has since been updated, is a recipe for disaster; Package {up,down}grades should only occur on an explicit basis. It is also not practical to assume that all of the clusters are able to move in version lock-step with one another. Ideally, our repository would be able to host multiple software versions, and Puppet could be utilized to ensure that the nodes of a cluster remained pinned to the apropos version. The approach cited above could be used here as well, but it would require the creation of a distribution for each new version added. Another approach would be to generate a packages manifest from a pool containing multiple versions, (and in the absence of pinning, hosts would simply get the most recent).

Event Timeline

fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a project: acl*sre-team.
fgiunchedi added subscribers: Aklapper, Matanya, fgiunchedi.

other examples include the nodejs upgrade in T107762: Switch RESTBase to use Node.js 4.2

Eevans renamed this task from consider debian "experimental" workflow for internal repository to Debian repository supporting multiple package versions.May 18 2016, 7:06 PM
Eevans updated the task description. (Show Details)

Previous related discussion:

An example of a repository manager with good support for multiple versions per package is aptly. I have used this in the past to host parsoid packages, and @yuvipanda is currently using it in labs.

This also affects Parsoid .deb package distribution. When we uploaded version 0.5.0 of the parsoid deb today, I noticed that the 0.4.1 version was removed. For now, this is not a problem, but, going forward, this is not desirable when we make breaking changes to Parsoid since not all versions of Parsoid, VE, and M/W are going to be compatible with each other. Without having multiple versions of a package, existing installs of VE won't be able to upgrade to a later version that is compatible unless they upgrade everything to remain compatible.

Perhaps another alternative, would be to lightly automate the creation of trivial APT repos using dpkg-scanpackages, for the select cases where this makes sense.

So using Cassandra as an example: You could have a repository for the current 2.2 packages, and another for the 2.1 packages, and template apt sources with Puppet accordingly.

deb jessie main non-free contrib
deb-src jessie main non-free contrib

# Current Cassandra 2.2.x release
deb 22x/

# Current Cassandra 2.1.x release
#deb 21x/

This would be really simple to do (I'm sure we could even find something that already does this).

[ ... ]
This would be really simple to do (I'm sure we could even find something that already does this).

And another option.

For both Nodepool and Zuul, I need packages versions that are not available in our For Nodepool that blocks further upgrades and for Zuul I went with a hack to get the Debian packaging to craft a virtualenv that fetches dependencies from pypi.

I would love to pin something like jessie-wikimedia/zuul which would get the Zuul package itself and all its dependencies with the proper versions.

MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff subscribed.

We have established that with the system of separate component we introduced a few years ago. Marking as resolved.