Page MenuHomePhabricator

CloudVPS: workaround archival of jessie-backports repo
Closed, ResolvedPublic

Description

We rely heavily on Debian jessie-backports repository to build both mitaka/jessie and mitaka/stretch servers.

I learned today that jessie-backports repo will be out of Debian mirrors network sooner than expected.

12:26 <moritzm> but with jessie being in LTS backports is closed
12:26 <moritzm> and will actually be removed once a Debian FTP master finds the time to do it
12:26 <moritzm> so puppet will fail on those
12:27 <moritzm> the more robust solution would have been to create a component/mitakajessie
12:27 <moritzm> import all the needed debs in there
12:27 <moritzm> and then pin the hosts to that component

I confirmed this by talking to Debian FTP/Backports masters:

12:35 <arturo> formorer: hi! just reading this https://lists.debian.org/debian-lts/2019/02/msg00101.html do you know of a tentative  date for jessie-backports archival? I'm using it a lot :-S
12:35 <jcristau> ~now
12:36 <formorer> yesterday
12:36 <formorer> expect it at every time

If the Debian jessie-backports repo disappears, puppet will fail in all current CloudVPS servers, and we won't be able to re-image servers.
There are a couple of ways to avoid the disaster:

  1. use the repo directly from the archive:
deb http://archive.debian.org/debian-archive/debian/ jessie-backports main
deb-src http://archive.debian.org/debian-archive/debian/ jessie-backports main
  1. import all the mitaka packages from Debian into the WMF reprepro as described in https://wikitech.wikimedia.org/wiki/Reprepro

While the first option is straight forward, and only involves a puppet patch to update the repo, in classes:

  • openstack::serverpackages::mitaka::jessie
  • openstack::serverpackages::mitaka::stretch
  • openstack::clientpackages::mitaka::jessie
  • openstack::clientpackages::mitaka::stretch

The second option allows us to retain the chance to patch packages if we need to. Will need the same puppet patch anyway to point to the new repo in apt.wikimedia.org.
This last approach is the one @MoritzMuehlenhoff recommended.

Event Timeline

aborrero renamed this task from CloudVPS: prevent archival of jessie-backports to CloudVPS: workaround archival of jessie-backports repo.Feb 19 2019, 11:58 AM
aborrero triaged this task as High priority.
aborrero created this task.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero updated the task description. (Show Details)Feb 19 2019, 12:01 PM

Change 491558 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: pull openstack mitaka packages into reprepro

https://gerrit.wikimedia.org/r/491558

Change 491736 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: use archive.debian.org as jessie-backports repo

https://gerrit.wikimedia.org/r/491736

Mentioned in SAL (#wikimedia-operations) [2019-02-21T12:00:59Z] <arturo> disable puppet in install1002 to test T216497

Mentioned in SAL (#wikimedia-operations) [2019-02-21T12:10:11Z] <arturo> T216497 import reprepro key 7638D0442B90D010 (debian archive automatic signing key (8/jessie)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T12:17:24Z] <arturo> enable puppet in install1002 (done testing T216497)

Change 491558 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: pull openstack mitaka packages into reprepro

https://gerrit.wikimedia.org/r/491558

Mentioned in SAL (#wikimedia-operations) [2019-02-21T12:23:55Z] <arturo> importing openstack mitaka packages to reprepro @ install1002 (T216497)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T12:33:33Z] <arturo> disable puppet in cloudnet2001-dev to test T216497

Note to self: these changes needs documentation in wikitech.

Change 491736 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: use new repository for mitaka packages

https://gerrit.wikimedia.org/r/491736

aborrero reopened this task as Open.Mar 12 2019, 1:17 PM

We need to drop all the -t jessie-backports things from the stretch hosts, now that the jessie-backports repo is not installed in them, otherwise we get an error by apt because we are referencing a repo we don't have configured.

Mentioned in SAL (#wikimedia-cloud) [2019-03-12T13:18:13Z] <arturo> T216497 create cloudnet-stretch-test-01 instance for testing puppet code

Change 495893 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: drop -t jessie-backports for openstack component installs in stretch

https://gerrit.wikimedia.org/r/495893

Change 495969 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: openstack-mitaka-jessie: don't import systemd

https://gerrit.wikimedia.org/r/495969

Change 495969 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: openstack-mitaka-jessie: don't import systemd

https://gerrit.wikimedia.org/r/495969

Mentioned in SAL (#wikimedia-operations) [2019-03-12T19:09:46Z] <arturo> T216497 manually delete systemd 230-7~bpo8+2 from jessie-wikimedia/openstack-mitaka-jessie

Mentioned in SAL (#wikimedia-operations) [2019-03-12T19:14:13Z] <arturo> T216497 manually delete libpam-systemd and libsystemd0 230-7~bpo8+2 from jessie-wikimedia/openstack-mitaka-jessie

Change 496156 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: virt: install libguestfs-tools package in component class

https://gerrit.wikimedia.org/r/496156

Change 496156 abandoned by Arturo Borrero Gonzalez:
openstack: virt: install libguestfs-tools package in component class

Reason:
This will be integrated into Change-Id: I7be898ef18a5724a1eb312efa23d17d983b192f8 instead

https://gerrit.wikimedia.org/r/496156

Change 495893 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: drop -t jessie-backports for openstack component installs in stretch

https://gerrit.wikimedia.org/r/495893

Mentioned in SAL (#wikimedia-operations) [2019-03-14T12:06:05Z] <arturo> T216497 drop some packages from jessie-wikimedia/openstack-mtiaka-jessie: libvirt*, librados2, librbd1, because they induce the resolver to conflict with those included in stretch

Change 496420 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: drop more packages from openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496420

Mentioned in SAL (#wikimedia-operations) [2019-03-14T12:12:24Z] <arturo> T216497 drop some packages from jessie-wikimedia/openstack-mtiaka-jessie: qemu-XXX

Change 496420 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: drop more packages from openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496420

Mentioned in SAL (#wikimedia-operations) [2019-03-14T13:10:37Z] <arturo> T216497 drop python-mysqldb from jessie-wikimedia/openstack-mtiaka-jessie

Mentioned in SAL (#wikimedia-operations) [2019-03-14T13:15:38Z] <arturo> T216497 drop libpulse0 from jessie-wikimedia/openstack-mtiaka-jessie

Change 496433 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: drop even more packages from jessie-wikimedia/openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496433

Change 496433 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: drop even more packages from jessie-wikimedia/openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496433

Change 496436 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] Revert "openstack: nova: mitaka: stretch: install python-dogpile.core from jessie"

https://gerrit.wikimedia.org/r/496436

Change 496436 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Revert "openstack: nova: mitaka: stretch: install python-dogpile.core from jessie"

https://gerrit.wikimedia.org/r/496436

Change 496476 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: drop python-dogpile.cache from openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496476

Mentioned in SAL (#wikimedia-operations) [2019-03-14T16:02:41Z] <arturo> T216497 drop python-dogpile.cache from jessie-wikimedia/openstack-mitaka-jessie

Change 496476 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: drop python-dogpile.cache from openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/496476

aborrero closed this task as Resolved.Mar 15 2019, 10:16 AM

We are apparently good for now.

Had revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/496863 because of a dependency issue on the baremetal servers running this kind of mixed setup.

Apparently, continuing the tradition of dependency hell, we have T218423 unable to proceed until a couple more python3 libraries are in our backports.

It looks like it's python3-cliff (as well as if any of the openstack client libraries in that patch if they are not there) is what is needed. I *think* that's the only one causing issues, though.

The following packages have unmet dependencies:
 python3-designateclient : Depends: python3-cliff (>= 1.15.0) but 1.7.0-2 is to be installed
E: Unable to correct problems, you have held broken packages.
Bstorm reopened this task as Open.Mar 15 2019, 9:24 PM
Bstorm closed this task as Resolved.

Never mind, I see that this just requires a pin to make it right. I'll note that on the other ticket.

Change 498058 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: include python-cliff into jessie-wikimedia/openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/498058

Change 498058 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: include python-cliff into jessie-wikimedia/openstack-mitaka-jessie

https://gerrit.wikimedia.org/r/498058

Mentioned in SAL (#wikimedia-operations) [2019-03-21T12:08:31Z] <arturo> T216497 add python-cliff to jessie-wikimedia/openstack-mitaka-jessie

Mentioned in SAL (#wikimedia-operations) [2019-03-21T12:40:16Z] <arturo> T216497 remove python-cliff from jessie-wikimedia/openstack-mitaka-jessie

jbond added a subscriber: jbond.Mar 21 2019, 4:26 PM

Change 500013 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: keystone: drop jessie-backport install option

https://gerrit.wikimedia.org/r/500013

Change 500013 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: keystone: drop jessie-backport install option

https://gerrit.wikimedia.org/r/500013

Mentioned in SAL (#wikimedia-operations) [2019-05-03T17:08:19Z] <arturo> T222148 drop libudev1 from openstack-mitaka-jessie/jessie-wikimedia (related to T216497)