Page MenuHomePhabricator

Locate an alternative source for modern Openstack Debian packages
Closed, ResolvedPublic

Description

Several services in Labs, tools, and prod rely on packages from the mirantis apt repo. That repo doesn't work, so those systems currently cannot be rebuilt.

Event Timeline

Andrew created this task.Jun 28 2017, 2:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 28 2017, 2:57 PM
Andrew triaged this task as High priority.Jun 28 2017, 2:58 PM
bd808 added a subscriber: bd808.Jun 28 2017, 10:37 PM

One bandaid we could try would be to gather up packages from /var/cache/apt/archives/ on our hosts and put them into an aptly repo. This is obviously a very temporary solution, but arguable better than not being able to rebuild a box if it goes down.

bd808 added a comment.Aug 3 2017, 3:39 PM

Jessie-backports currently has Mitaka package versions. As noted by Mortiz on irc it is unlikely that those packages will be updated again now that stretch is released. There is a bit of scariness in not having an easy way to pin our installed packages to the mitaka versions like the Mirantis repos did, but its probably something we can survive.

Change 369957 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: Jessie has Liberty py3 client libraries

https://gerrit.wikimedia.org/r/369957

Change 369957 merged by Rush:
[operations/puppet@production] openstack: Jessie has Liberty py3 client libraries

https://gerrit.wikimedia.org/r/369957

Change 369972 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: Jessie has cannot fulfill py3 clients without backports

https://gerrit.wikimedia.org/r/369972

Change 369972 merged by Rush:
[operations/puppet@production] openstack: Jessie has cannot fulfill py3 clients without backports

https://gerrit.wikimedia.org/r/369972

Change 369987 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: revert shenkengen to py2 libs

https://gerrit.wikimedia.org/r/369987

Change 369987 merged by Madhuvishy:
[operations/puppet@production] openstack: revert shenkengen to py2 libs

https://gerrit.wikimedia.org/r/369987

chasemp added a subscriber: chasemp.Feb 9 2018, 6:35 PM

In looking at a total move to Debian as solution there is at least one glaring issue in that the cadence of stable=>backports(testing) from next release does not seem to follow a N+1 release cycle. For instance Jessie=>Stretch=>Buster is poised to miss Ocata entirely. That's a pretty significant hurdle if it's sane for us at all.

Notes from investigating:


OpenStack Upgrade release policy states

We only test N to N+1 in the CI system (the grenade jobs) so while
things might work in practice by skipping a release, it's not tested
 that way upstream at all so any chance of it working is accidental and
shouldn't be relied upon or really expected to be supported that way upstream.

A the moment it sure looks like there is no way to avoid a troubling and expected faulty version jump from jessie=>stretch=>buster (ocata is missing)


https://releases.openstack.org/teams/keystone.html
https://wiki.openstack.org/wiki/Releases
http://snapshot.debian.org/
https://wiki.debian.org/DebianReleases#Production_Releases
https://wiki.debian.org/LTS

https://qa.debian.org/developer.php?email=openstack-devel%40lists.alioth.debian.org <--- package version matrix
http://snapshot.debian.org/package/keystone/ <--snapshot for keystone

Openstack release(s) by codename

v9 == mitaka
v10 == newton
v11 == ocata
v12 == pike

Jessie Stable: (Icehouse)
jessie (oldstable) (python): OpenStack identity service
2014.1.3-6: all

Jessie Backports: (Mitaka) https://releases.openstack.org/mitaka/index.html
jessie-backports (python): OpenStack identity service
2:9.0.0-2~bpo8+1: all

Debian 9 “Stretch” LTS is targeted at 2020 to June 2022

Stretch Stable: (Newton) https://releases.openstack.org/newton/index.html
stretch (stable) (python): OpenStack identity service
2:10.0.0-9: all

Stretch Backports:

  • None

Buster Stable:(Pike) https://releases.openstack.org/pike/index.html
buster (testing) (python): OpenStack identity service
2:12.0.0-2: all
Buster Backports:

Andrew added a comment.Feb 9 2018, 8:57 PM

I've been assuming that we need to move to source-based deployments since Debian seems determined to do this the wrong way. I've contacted the debian packager (Allison Randal) several times about this but haven't gotten any useful responses.

Skipping versions seems semi-possible if we have to, but given the current cadence I don't have a lot of faith in how they'll handle this going forward.

aborrero added a subscriber: aborrero.EditedFeb 13 2018, 12:39 PM

I can add a bit more details.

TL;DR:

  • Ocata (v11) was not packaged for Debian on purpose
  • There are 2 options for using Ocata in Debian: build ourselves the packages OR use the ultimum.io repo

13:25 <arturo> DebianDeveloper: did you read my message regarding keystone v10 --> v11 --> v12?
13:26 <DebianDeveloper> arturo: Not sure, can you say again?
13:27 <arturo> sure
13:27 <arturo> DebianDeveloper: http://snapshot.debian.org/package/keystone/ <--- there seem to be no upgrade path for keystone in debian, right?
13:27 <DebianDeveloper> arturo: I skipped packaging Ocata, indeed, but we do have the source code available in the Git.
13:28 <DebianDeveloper> So you "just" need to rebuild it from the Git.
13:28 <arturo> I see
13:28 <arturo> was there any technical reason for the skip?
13:28 <DebianDeveloper> There's also a repository with backports for Ocata in Stretch done by the CZ people.
13:29 <arturo> oh, could you share the link?
13:29 <DebianDeveloper> arturo: There's a social one: I got out of employer, and was a bit demotivated, plus I expected others would pick-up, which never happened.
13:29 <arturo> DebianDeveloper: ok, understood
13:30 <DebianDeveloper> arturo: repo : deb http://debian.ultimum.io debian-stretch ocata ocata-backports
13:30 <DebianDeveloper> apt_key : http://debian.ultimum.io/apt.key
13:30 <arturo> thanks!
13:31 <DebianDeveloper> arturo: I thought about publishing that on a debian.net repo, but so far, I didn't.
13:31 <arturo> that would be great, indeed
13:32 <DebianDeveloper> arturo: If I was to publish Pike and Queens on official Debian backports, would you use it?
13:32 <DebianDeveloper> That's about 3/4 days of work, which is why I don't want to do it for nothing.
13:33 <arturo> DebianDeveloper: I fully understand the think about sharing efforts et al
13:34 <arturo> DebianDeveloper: about Pike/Queens, we are currently evaluating in the team how to do new deployments
13:34 <arturo> I bet for using Debian, that's for sure, but others in my team prefer source deployments
13:35 <arturo> (one of the points being the upgrade path thing)
13:39 <DebianDeveloper> arturo: If using sources, it means you're re-inventing all what's in the package maintainer's scripts. That's a huge amount of work.
13:40 <arturo> I see
13:42 <arturo> from the resources point of view, I guess there is little we could do from my team, given we are a non-profit foundation, fully donation-based income. Our resources are very limited. I wonder what happens with the big company players out there

I've checked the ultimum.io repo by hand and it's full of packages, apparently ready to use.

Andrew added a comment.Jul 6 2018, 9:29 PM

This is pretty much resolved thanks to various gymnastics with upstream debian repos. We're currently using jessie-backports for Mitaka.

@chasemp or @aborrero could you note here what repos we plan to use for the next few OpenStack releases? Then this can be closed.

According to my sources, Openstack Newton (mitaka + 1) is in Debian stretch standard repos. All releases after Newton (ocata, etc) are probably located in stretch-backports

This should contain ocata:

deb http://debian.ultimum.io debian-stretch ocata
aborrero closed this task as Resolved.Oct 18 2018, 9:42 AM
aborrero claimed this task.

So, conclusions:

  • openstack mitaka is in jessie-backports (already using it in eqiad1)
  • openstack newton is in stretch (mitaka+1)
  • openstack ocata is in stretch-backports (newton+1)

Closing task now.

If you are looking at this bug report, is probably because you are looking for this information: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_source