One bandaid we could try would be to gather up packages from /var/cache/apt/archives/ on our hosts and put them into an aptly repo. This is obviously a very temporary solution, but arguable better than not being able to rebuild a box if it goes down.

Jessie-backports currently has Mitaka package versions. As noted by Mortiz on irc it is unlikely that those packages will be updated again now that stretch is released. There is a bit of scariness in not having an easy way to pin our installed packages to the mitaka versions like the Mirantis repos did, but its probably something we can survive.

Change 369957 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: Jessie has Liberty py3 client libraries

https://gerrit.wikimedia.org/r/369957

Change 369957 merged by Rush:
[operations/puppet@production] openstack: Jessie has Liberty py3 client libraries

https://gerrit.wikimedia.org/r/369957

Change 369972 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: Jessie has cannot fulfill py3 clients without backports

https://gerrit.wikimedia.org/r/369972

Change 369972 merged by Rush:
[operations/puppet@production] openstack: Jessie has cannot fulfill py3 clients without backports

https://gerrit.wikimedia.org/r/369972

Change 369987 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: revert shenkengen to py2 libs

https://gerrit.wikimedia.org/r/369987

Change 369987 merged by Madhuvishy:
[operations/puppet@production] openstack: revert shenkengen to py2 libs

https://gerrit.wikimedia.org/r/369987

In looking at a total move to Debian as solution there is at least one glaring issue in that the cadence of stable=>backports(testing) from next release does not seem to follow a N+1 release cycle. For instance Jessie=>Stretch=>Buster is poised to miss Ocata entirely. That's a pretty significant hurdle if it's sane for us at all.

Notes from investigating:

OpenStack Upgrade release policy states

We only test N to N+1 in the CI system (the grenade jobs) so while
things might work in practice by skipping a release, it's not tested
 that way upstream at all so any chance of it working is accidental and
shouldn't be relied upon or really expected to be supported that way upstream.

A the moment it sure looks like there is no way to avoid a troubling and expected faulty version jump from jessie=>stretch=>buster (ocata is missing)

https://releases.openstack.org/teams/keystone.html
https://wiki.openstack.org/wiki/Releases
http://snapshot.debian.org/
https://wiki.debian.org/DebianReleases#Production_Releases
https://wiki.debian.org/LTS

https://qa.debian.org/developer.php?email=openstack-devel%40lists.alioth.debian.org <--- package version matrix
http://snapshot.debian.org/package/keystone/ <--snapshot for keystone

Openstack release(s) by codename

v9 == mitaka
v10 == newton
v11 == ocata
v12 == pike

Jessie Stable: (Icehouse)
jessie (oldstable) (python): OpenStack identity service
2014.1.3-6: all

Jessie Backports: (Mitaka) https://releases.openstack.org/mitaka/index.html
jessie-backports (python): OpenStack identity service
2:9.0.0-2~bpo8+1: all

Debian 9 “Stretch” LTS is targeted at 2020 to June 2022

Stretch Stable: (Newton) https://releases.openstack.org/newton/index.html
stretch (stable) (python): OpenStack identity service
2:10.0.0-9: all

Stretch Backports:

• None

Buster Stable:(Pike) https://releases.openstack.org/pike/index.html
buster (testing) (python): OpenStack identity service
2:12.0.0-2: all
Buster Backports:

I've been assuming that we need to move to source-based deployments since Debian seems determined to do this the wrong way. I've contacted the debian packager (Allison Randal) several times about this but haven't gotten any useful responses.

Skipping versions seems semi-possible if we have to, but given the current cadence I don't have a lot of faith in how they'll handle this going forward.

I can add a bit more details.

TL;DR:

• Ocata (v11) was not packaged for Debian on purpose
• There are 2 options for using Ocata in Debian: build ourselves the packages OR use the ultimum.io repo

13:25 <arturo> DebianDeveloper: did you read my message regarding keystone v10 --> v11 --> v12?
13:26 <DebianDeveloper> arturo: Not sure, can you say again?
13:27 <arturo> sure
13:27 <arturo> DebianDeveloper: http://snapshot.debian.org/package/keystone/ <--- there seem to be no upgrade path for keystone in debian, right?
13:27 <DebianDeveloper> arturo: I skipped packaging Ocata, indeed, but we do have the source code available in the Git.
13:28 <DebianDeveloper> So you "just" need to rebuild it from the Git.
13:28 <arturo> I see
13:28 <arturo> was there any technical reason for the skip?
13:28 <DebianDeveloper> There's also a repository with backports for Ocata in Stretch done by the CZ people.
13:29 <arturo> oh, could you share the link?
13:29 <DebianDeveloper> arturo: There's a social one: I got out of employer, and was a bit demotivated, plus I expected others would pick-up, which never happened.
13:29 <arturo> DebianDeveloper: ok, understood
13:30 <DebianDeveloper> arturo: repo : deb http://debian.ultimum.io debian-stretch ocata ocata-backports
13:30 <DebianDeveloper> apt_key : http://debian.ultimum.io/apt.key
13:30 <arturo> thanks!
13:31 <DebianDeveloper> arturo: I thought about publishing that on a debian.net repo, but so far, I didn't.
13:31 <arturo> that would be great, indeed
13:32 <DebianDeveloper> arturo: If I was to publish Pike and Queens on official Debian backports, would you use it?
13:32 <DebianDeveloper> That's about 3/4 days of work, which is why I don't want to do it for nothing.
13:33 <arturo> DebianDeveloper: I fully understand the think about sharing efforts et al
13:34 <arturo> DebianDeveloper: about Pike/Queens, we are currently evaluating in the team how to do new deployments
13:34 <arturo> I bet for using Debian, that's for sure, but others in my team prefer source deployments
13:35 <arturo> (one of the points being the upgrade path thing)
13:39 <DebianDeveloper> arturo: If using sources, it means you're re-inventing all what's in the package maintainer's scripts. That's a huge amount of work.
13:40 <arturo> I see
13:42 <arturo> from the resources point of view, I guess there is little we could do from my team, given we are a non-profit foundation, fully donation-based income. Our resources are very limited. I wonder what happens with the big company players out there

I've checked the ultimum.io repo by hand and it's full of packages, apparently ready to use.

This is pretty much resolved thanks to various gymnastics with upstream debian repos. We're currently using jessie-backports for Mitaka.

@chasemp or @aborrero could you note here what repos we plan to use for the next few OpenStack releases? Then this can be closed.

According to my sources, Openstack Newton (mitaka + 1) is in Debian stretch standard repos. All releases after Newton (ocata, etc) are probably located in stretch-backports

This should contain ocata:

deb http://debian.ultimum.io debian-stretch ocata

If you are looking at this bug report, is probably because you are looking for this information: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_source

The matrix suggests that Ocata packages are available in stretch-backports. I don't see that, though... were they removed, or am I looking in the wrong place?

root@cloudcontrol1003:~# apt-get update
Ign:1 http://mirrors.wikimedia.org/debian stretch InRelease
Hit:2 http://mirrors.wikimedia.org/debian stretch-updates InRelease
Get:3 http://mirrors.wikimedia.org/debian stretch-backports InRelease [91.8 kB]
Hit:4 http://apt.wikimedia.org/wikimedia stretch-wikimedia InRelease
Hit:6 http://mirrors.wikimedia.org/debian stretch Release                             
Hit:5 http://security-cdn.debian.org/debian-security stretch/updates InRelease
Fetched 91.8 kB in 1s (55.2 kB/s)
No directory, logging in with HOME=/
INFO:debmonitor:Found 949 installed binary packages
INFO:debmonitor:Found 7 upgradable binary packages (including new dependencies)
INFO:debmonitor:Successfully sent the upgradable update to the DebMonitor server
Reading package lists... Done
root@cloudcontrol1003:~# apt-cache madison nova-compute
nova-compute | 2:14.0.0-4+deb9u1 | http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
nova-compute | 2:14.0.0-4+deb9u1 | http://security.debian.org/debian-security stretch/updates/main amd64 Packages
      nova | 2:14.0.0-4+deb9u1 | http://mirrors.wikimedia.org/debian stretch/main Sources
      nova | 2:14.0.0-4+deb9u1 | http://security.debian.org/debian-security stretch/updates/main Sources