So labstore1002 (current active labstore host) has jessie-backports enabled, and labstore1001 and other hosts don't. This causes puppet failures since we want python3-pymysql, which is present only in jessie-backports.
Options:
Either way, these should all be uniform across the cluster.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| apt: enable backports on Debian systems | operations/puppet | production | +10 -19 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T105720 Labs team reliability goal for Q1 2015/16 | |||
| Resolved | coren | T106479 Ensure that labstore machine is 'known good' hardware | |||
| Resolved | coren | T107038 Switch NFS server back to labstore1001 | |||
| Resolved | coren | T102478 labstore has multiple unpuppetized files/scripts/configs | |||
| Resolved | coren | T107574 Reinstall labstore1001 and make sure everything is puppet-ready | |||
| Resolved | faidon | T107507 Investigate whether to use Debian's jessie-backports | |||
| Declined | faidon | T108941 Make certain that jessie-backports is disabled fleetwide. |
I'd rather enable jessie-backports, I think, since the packages there will be maintained actively by the community; and they are deactivated by default.
Option c: Enable jessie-backports only for the hosts that need it and only for the package that is needed.
I've just synced the sources for both of them, and puppet succeeds. Leaving this open to determine the right thing to do...
If the apt pinning for backports is configured in a way that backports is only selected on a case-by-case basis by running e.g. "apt-get install -t jessie-backports install foo", this seems fine to me. jessie-backports is part of the official Debian infrastructure these days (i..e also protected by Secure Apt etc) and if we ever make jessie-backports on our own in the future, we can push them to jessie-backports as well.
Compared to the standard archive backports has a number of drawbacks, so we should use these carefully:
The difference between the two labstores is because of a change in upstream d-i during the jessie RC cycle that disabled backports by default, cf. Debian #764982. I expect the rest of the fleet to be in a similar inconsistent state depending on the date they were installed.
I don't think there is, no. We can pick a version manually but that sucks; we can also set up separate pins per package, which is also not great.
We could write a Package provider that is just a tweak of the apt-get one and use 'provider => "backports"'
(I'd prefer the pinning approach because it's easier to read and I intend to use it in the future for Toolforge execution instances, but:)
If backports is only needed for python3-pymysql, can't we use a MySQL Python library that ships with Jessie and rewrite the one (1) script (modules/labstore/files/create-dbusers) that depends on it? (There is another reference in modules/nodepool/templates/nodepool.yaml.erb, but I'm sure that could be worked around as well.)
@scfc Right now, we have one clearly used package from it but we've used backports (in general, not necessarily from a repo) before in production and in Labs so I think the question remains valid for the general case, that is - whether we want to enable Jessie's backports and how to get packages from them if we do.
Opsen consensus on IRC is that jessie-backports should be disabled fleet-wide and any needed package brought into jessie-wikimedia.
(I submitted a change for Toolforge to enable backports for specific packages only at https://gerrit.wikimedia.org/r/#/c/238662/.)
Sorry, if I recall correctly most of it was held on an unlogged channel. The discussion at heart, however, is simple enough: -backports is felt to be less robust (especially for security updates) and given that the number of packages we effectively pull from it is small, it was felt that the overhead of keeping those packages into our own repos was preferable since it allows us to keep a close eye on them.
I don't remember this IRC discussion. Who was attending it? A little more context please? :)
In any case, I disagree with that consensus. I think enabling backports fleet-wide (and only for Debian, not Ubuntu) is fine and preferrable to us copying files across all the time, as long we do so carefully and more importantly consciously. Enabling it doesn't mean preferring packages from there, as backports are set to NotAutomatic: yes (and ButWithAutomaticUpdates: yes), which means they're pinned with priority 100.
There is no good way to do so with puppet but I think the usage is so limited so far that we don't have to have a huge discussion about this yet. I don't think we should use it for important components of the infrastructure like e.g. Apache, but I think for e.g. a few Python libraries here and there it's going to be just fine.
Any objections? If so, please explain why, a reference to an unlogged discussion alone doesn't help. @MoritzMuehlenhoff, what do you think?
@faidon: That was mostly you and Moritz. Lemme see if I find quotables in my local logs. :-)
Using packages from backport selectively is fine with me, we already do it e.g. with openjdk-8 which we need for the cassandra cluster. It's a valid part of the Debian packaging ecosystem.
I only remember the discussion back then vaguely, but from what I remember the concern was puppet's poor support (or maybe rather an uncertaintly how good it will work out in practice) for a setup where some packages are selected from backports. If that is tested/resolved, it seems fine with me.
Change 252202 had a related patch set uploaded (by Faidon Liambotis):
apt: enable backports on Debian systems
Some systems already had the jessie backports repo in sources.list, I think it varies depending on the install date of the system. Will probably have to clean up with salt.
I very welcome the decision to add jessie-backports. That has hit me when setting up Nodepool, the POC had jessie-backports but when we rebuild it we had to import a lot of dependencies into our jessie-wikimedia/backports :-/
A random example is python-novaclient:
100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages 1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
We would probably want to clean up jessie-wikimedia/backports of duplicate packages.