Page MenuHomePhabricator

Handle sunset of stretch-backports
Open, HighPublic

Description

backports is only available for Debian stable and oldstable and Debian stretch will move to LTS state in about a week. Packages which we were using from stretch-backports need to be migrate to components on apt.wikimedia.org.

  • Stretch Docker base image
  • librdkafka1 - eventlogging
  • osm2pgsql - osm
  • analytics - git-lfs
  • python3-prometheus-client-package - prometheus
  • systemd, systemd-sysv, udev, libsystemd0, libpam-systemd - systemd
  • npm - testreduce
  • libhwloc5 - trafficserver
  • librdkafka1 - varnishkafka
  • graphite-web - graphite
  • npm - testreduce T257906
  • lilypond - mediawiki

Non exhaustive list for CI (at least)

  • golang-1.11-go - CI tox-acme-chef image T257456
  • golang-1.11-go - docker-registry.wikimedia.org/golang:1.11 which is based on Stretch
  • git 2.20.1 - git-protocol-v2 for CI and scap at least. T257308

These are handled via T256881

  • 'librados2', 'librgw2', 'librbd1', 'python-rados', 'python-rbd', 'ceph-common', 'python-cephfs', 'libradosstriper1' - openstack/queens&rocky

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2020, 2:04 PM
hashar added a subscriber: hashar.Jul 1 2020, 6:35 PM

From a puppet codesearch, I see:

  • librdkafka1 - eventlogging
  • 'librados2', 'librgw2', 'librbd1', 'python-rados', 'python-rbd', 'ceph-common', 'python-cephfs', 'libradosstriper1' - openstack/queens&rocky
  • osm2pgsql - osm
  • analytics - git-lfs
  • python3-prometheus-client-package - prometheus
  • systemd, systemd-sysv, udev, libsystemd0, libpam-systemd - systemd
  • npm - testreduce
  • libhwloc5 - trafficserver
  • librdkafka1 - varnishkafka

Change 609100 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove obsolete pinning for liblwloc

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609100

Change 609104 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Unconditionally install systemd packages

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609104

hashar added a comment.Jul 2 2020, 8:10 AM

I am wondering how to find packages that got installed from stretch-backports. Our base images have backports configured and one can borrow a package from it when it does not exist in stretch (an example is golang-1.11-go) or by specifying an explicit version pinning.

I am wondering how to find packages that got installed from stretch-backports.

Any package from backports should follow the "~bpo9+X" naming convention, so you should be able to find affected package versions in debmonitor.wikimedia.org

Change 609100 merged by Muehlenhoff:
[operations/puppet@production] Remove obsolete pinning for libhwloc

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609100

Change 609181 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Create component/systemd241

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609181

Change 609181 merged by Muehlenhoff:
[operations/puppet@production] Create component/systemd241

https://gerrit.wikimedia.org/r/609181

Change 609419 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] systemd/slice: Install systemd 241 from component/systemd241

https://gerrit.wikimedia.org/r/609419

Change 609104 abandoned by Muehlenhoff:
[operations/puppet@production] Unconditionally install systemd packages

Reason:
In favour of https://gerrit.wikimedia.org/r/c/operations/puppet/ /609419/

https://gerrit.wikimedia.org/r/609104

Change 609420 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove apt::pin for python3-prometheus-client-package

https://gerrit.wikimedia.org/r/609420

Change 609420 merged by Elukey:
[operations/puppet@production] Remove apt::pin for python3-prometheus-client-package

https://gerrit.wikimedia.org/r/609420

@MoritzMuehlenhoff I'm now thinking this is going to happen with every single debian release (archival of the backports repo).

Perhaps a more sustainable way to approach this is to mirror the backports repo ourselves and drop a whole category of problems in the package management field.

Change 609419 merged by Muehlenhoff:
[operations/puppet@production] systemd/slice: Install systemd 241 from component/systemd241

https://gerrit.wikimedia.org/r/609419

Change 610015 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop installing git-lfs from stretch-backports

https://gerrit.wikimedia.org/r/610015

hashar updated the task description. (Show Details)Tue, Jul 7, 12:20 PM

Change 610043 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Drop Puppet code which tries to install graphite-web from stretch-bpo

https://gerrit.wikimedia.org/r/610043

Change 610050 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop including backports in Stretch images

https://gerrit.wikimedia.org/r/610050

Change 610043 merged by Muehlenhoff:
[operations/puppet@production] Drop Puppet code which tries to install graphite-web from stretch-bpo

https://gerrit.wikimedia.org/r/610043

Change 610120 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove obsolete apt::pin for librdkafka1 on eventlogging

https://gerrit.wikimedia.org/r/610120

Change 610120 merged by Muehlenhoff:
[operations/puppet@production] Remove obsolete apt::pin for librdkafka1 on eventlogging

https://gerrit.wikimedia.org/r/610120

@MoritzMuehlenhoff I'm now thinking this is going to happen with every single debian release (archival of the backports repo).

Perhaps a more sustainable way to approach this is to mirror the backports repo ourselves and drop a whole category of problems in the package management field.

Yes, that is a recurring issue, but mirroring something which intentionally vanishes isn't a solution. backports is explicitly designed as the stepstone to the next release, it's not a PPA-like mechanism or a library of old/abandoned code. Most of the patches merged for prod were actually dropping unused pins since the underlying service had moved to Buster in the mean time.

Change 610015 merged by Muehlenhoff:
[operations/puppet@production] Stop installing git-lfs from stretch-backports

https://gerrit.wikimedia.org/r/610015

Mentioned in SAL (#wikimedia-operations) [2020-07-09T08:23:46Z] <moritzm> imported osm2pgsql 0.96.0+ds-1~bpo9+1 to "main" component T256877

Change 610706 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] No longer install osm2pgsql from stretch-backports

https://gerrit.wikimedia.org/r/610706

Change 610706 merged by Muehlenhoff:
[operations/puppet@production] No longer install osm2pgsql from stretch-backports

https://gerrit.wikimedia.org/r/610706

@MoritzMuehlenhoff I'm now thinking this is going to happen with every single debian release (archival of the backports repo).

Perhaps a more sustainable way to approach this is to mirror the backports repo ourselves and drop a whole category of problems in the package management field.

Yes, that is a recurring issue, but mirroring something which intentionally vanishes isn't a solution. backports is explicitly designed as the stepstone to the next release, it's not a PPA-like mechanism or a library of old/abandoned code. Most of the patches merged for prod were actually dropping unused pins since the underlying service had moved to Buster in the mean time.

I think I understand your point. Cleanup is great, indeed. But not all we are doing is cleanup. Having a mirror or replica would relax pressure on us generated by external parties. I mean, this already happened in the past and can put a lot of pressure.

Having a mirror so we can decide when to do such cleanup and upgrading is something really interesting to consider.

Change 612568 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove apt pin for stretch-backports for npm

https://gerrit.wikimedia.org/r/612568

Change 612568 merged by Muehlenhoff:
[operations/puppet@production] Remove apt pin for stretch-backports for npm

https://gerrit.wikimedia.org/r/612568

An exhaustive list of docker registry images that have bpo9 packages is at P11925 (alongside the script that generated them)

Change 613611 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Disable backports on stretch

https://gerrit.wikimedia.org/r/613611

Change 616077 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add component/lilypond

https://gerrit.wikimedia.org/r/616077

Change 616077 merged by Muehlenhoff:
[operations/puppet@production] Add component/lilypond

https://gerrit.wikimedia.org/r/616077

MoritzMuehlenhoff triaged this task as High priority.Fri, Jul 24, 1:35 PM
MoritzMuehlenhoff updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2020-07-31T07:50:20Z] <moritzm> uploaded lilypond 2.19.81+really-2.18.2-13~bpo9+1+wmf1 to stretch-wikimedia T256877