Page MenuHomePhabricator

Add python 3 packages to openstack::clientpackages::common
Open, Needs TriagePublicBUG REPORT

Description

So since https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495757/, all Cloud VPS VMs have included openstack clientpackages
And this has been fine as the first user of this (across all VMs) is in Python 2. However, Python 3 packages are missing, and at some point ideally this year, that script (among others) would be ported to the new version...

Details

Event Timeline

Krenair created this task.Mar 15 2019, 7:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2019, 7:05 PM

Change 496863 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] openstack::clientpackages::common: include python3 packages

https://gerrit.wikimedia.org/r/496863

Krenair claimed this task.Mar 15 2019, 7:13 PM

Change 496863 merged by Bstorm:
[operations/puppet@production] openstack::clientpackages::common: include python3 packages

https://gerrit.wikimedia.org/r/496863

Bstorm added a subscriber: Bstorm.Mar 15 2019, 9:28 PM

Ok, it looks like to make that patch work correctly on our heavily-pinned Jessie-Mitaka setup, the python3 packages need to be pinned properly in modules/openstack/manifests/clientpackages/mitaka/jessie.pp

Change 497009 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Re-apply "openstack::clientpackages::common: include python3 packages"

https://gerrit.wikimedia.org/r/497009

One thing we'll need to figure out is this:

krenair@krenair-clientpackages-py3-jessie:~$ apt-cache policy python3-cliff
python3-cliff:
  Installed: 1.15.0-4~bpo8+1
  Candidate: 1.15.0-4~bpo8+1
  Package pin: 1.15.0-4~bpo8+1
  Version table:
 *** 1.15.0-4~bpo8+1 1002
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     1.7.0-2 1002
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

That 1.15 version we need only comes from the jessie-backports mirror, but isn't that supposed to be gone now?

One thing we'll need to figure out is this:

krenair@krenair-clientpackages-py3-jessie:~$ apt-cache policy python3-cliff
python3-cliff:
  Installed: 1.15.0-4~bpo8+1
  Candidate: 1.15.0-4~bpo8+1
  Package pin: 1.15.0-4~bpo8+1
  Version table:
 *** 1.15.0-4~bpo8+1 1002
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     1.7.0-2 1002
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

That 1.15 version we need only comes from the jessie-backports mirror, but isn't that supposed to be gone now?

Correct. Sorry, I gave you review in a patch without reading this phab task first.

We now have a jessie-wikimedia/openstack-mitaka-jessie component in the apt.wikimedia.org repo, and we may include this package there. However, I would advice to carefully review which version of python-cliff we need, because adding packages there have consequences. Among other things, a package added to that component repo will be available (and likely installed) to Debian Stretch servers, which may cause conflicts with the own python-cliff version in the Debian stretch official repo.

Well python3-designateclient and python3-neutronclient in jessie and stretch all depend on python3-cliff (>= 1.15.0). Stretch has 1.15.0-5 by default (500) and 2.13.0-1~bpo9+2 in stretch-backports (100).
Why is it that stretch servers have this repo though? I mean I know it would go from serverpackages::mitaka::stretch through commonpackages::mitaka and pick up the Apt::Repository['openstack-mitaka-jessie'] there, but fundamentally, applying jessie packages on a stretch server? Shouldn't they be separated, so we can do stuff like this without worrying too much?
What priority would a stretch server assign a python3-cliff package if it got uploaded to openstack-mitaka-jessie? I would hope that if it is 500 or above the package is similar enough to the default stretch one to not cause issues - https://metadata.ftp-master.debian.org/changelogs/main/p/python-cliff/python-cliff_1.15.0-5_changelog vs. https://metadata.ftp-master.debian.org/changelogs/main/p/python-cliff/python-cliff_1.15.0-4~bpo8+1_changelog makes it look like they just changed how debian deals with the openstack upstream a bit?

Why is it that stretch servers have this repo though?

It is an intermediate step in upgrading OpenStack from mitaka to newton across the cluster (T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton).

Krenair added a comment.EditedMar 19 2019, 11:56 PM
alex@alex-laptop:~/Development/Wikimedia/T218423$ wget  http://mirrors.wikimedia.org/debian/pool/main/p/python-cliff/python3-cliff_1.15.0-4~bpo8+1_all.deb
--2019-03-19 23:50:08--  http://mirrors.wikimedia.org/debian/pool/main/p/python-cliff/python3-cliff_1.15.0-4~bpo8+1_all.deb
Resolving mirrors.wikimedia.org (mirrors.wikimedia.org)... 2620:0:861:1:208:80:154:15, 208.80.154.15
Connecting to mirrors.wikimedia.org (mirrors.wikimedia.org)|2620:0:861:1:208:80:154:15|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24044 (23K) [application/octet-stream]
Saving to: ‘python3-cliff_1.15.0-4~bpo8+1_all.deb’

python3-cliff_1.15.0-4~bpo8+1_all.deb              100%[================================================================================================================>]  23.48K  --.-KB/s    in 0.09s   

2019-03-19 23:50:08 (258 KB/s) - ‘python3-cliff_1.15.0-4~bpo8+1_all.deb’ saved [24044/24044]

alex@alex-laptop:~/Development/Wikimedia/T218423$ wget http://cdn-fastly.deb.debian.org/debian/pool/main/p/python-cliff/python3-cliff_1.15.0-5_all.deb
--2019-03-19 23:50:13--  http://cdn-fastly.deb.debian.org/debian/pool/main/p/python-cliff/python3-cliff_1.15.0-5_all.deb
Resolving cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)... 2a04:4e42:f::204, 151.101.60.204
Connecting to cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)|2a04:4e42:f::204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23704 (23K) [application/x-debian-package]
Saving to: ‘python3-cliff_1.15.0-5_all.deb’

python3-cliff_1.15.0-5_all.deb                     100%[================================================================================================================>]  23.15K  --.-KB/s    in 0.01s   

2019-03-19 23:50:13 (1.98 MB/s) - ‘python3-cliff_1.15.0-5_all.deb’ saved [23704/23704]

alex@alex-laptop:~/Development/Wikimedia/T218423$ ls
python3-cliff_1.15.0-4~bpo8+1_all.deb  python3-cliff_1.15.0-5_all.deb
alex@alex-laptop:~/Development/Wikimedia/T218423$ dpkg-deb -R python3-cliff_1.15.0-4~bpo8+1_all.deb tmp-jessie
alex@alex-laptop:~/Development/Wikimedia/T218423$ dpkg-deb -R python3-cliff_1.15.0-5_all.deb tmp-stretch
alex@alex-laptop:~/Development/Wikimedia/T218423$ diff -r tmp-jessie tmp-stretch
diff -r tmp-jessie/DEBIAN/control tmp-stretch/DEBIAN/control
3c3
< Version: 1.15.0-4~bpo8+1
---
> Version: 1.15.0-5
6c6
< Installed-Size: 143
---
> Installed-Size: 128
diff -r tmp-jessie/DEBIAN/md5sums tmp-stretch/DEBIAN/md5sums
5c5
< fc3eba2f33409a29b174ed405fb2e4d3  usr/lib/python3/dist-packages/cliff-1.15.0.egg-info/requires.txt
---
> d41d8cd98f00b204e9800998ecf8427e  usr/lib/python3/dist-packages/cliff-1.15.0.egg-info/requires.txt
41c41
< 2d549edfe971ae82d87fd81af4dc7093  usr/share/doc/python3-cliff/changelog.Debian.gz
---
> 5983c2637e7a62273980a0307c117950  usr/share/doc/python3-cliff/changelog.Debian.gz
diff -r tmp-jessie/DEBIAN/postinst tmp-stretch/DEBIAN/postinst
4c4
< # Automatically added by dhpython:
---
> # Automatically added by dh_python3:
diff -r tmp-jessie/DEBIAN/prerm tmp-stretch/DEBIAN/prerm
4c4
< # Automatically added by dhpython:
---
> # Automatically added by dh_python3:
diff -r tmp-jessie/usr/lib/python3/dist-packages/cliff-1.15.0.egg-info/requires.txt tmp-stretch/usr/lib/python3/dist-packages/cliff-1.15.0.egg-info/requires.txt
1,8d0
< pbr<2.0,>=1.4
< cmd2>=0.6.7
< PrettyTable<0.8,>=0.7
< pyparsing>=2.0.1
< six>=1.9.0
< stevedore>=1.5.0
< unicodecsv>=0.8.0
< PyYAML>=3.1.0
Binary files tmp-jessie/usr/share/doc/python3-cliff/changelog.Debian.gz and tmp-stretch/usr/share/doc/python3-cliff/changelog.Debian.gz differ

I wonder if that stretch package is *supposed* to have an empty /usr/lib/python3/dist-packages/cliff-1.15.0.egg-info/requires.txt...

Change 497009 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Alex Monk):
[operations/puppet@production] Re-apply "openstack::clientpackages::common: include python3 packages"

https://gerrit.wikimedia.org/r/497009

Change 497009 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Re-apply "openstack::clientpackages::common: include python3 packages"

https://gerrit.wikimedia.org/r/497009

aborrero added a comment.EditedMar 21 2019, 12:48 PM

As I was suspecting from the beginning this is not that easy. We have complex dependency chains.

When trying to install the py3 version of the clientpackages in the hardware servers (mostly mitaka/stretch) I found this:

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-novaclient' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3-novaclient : Depends: python3-oslo.serialization (>= 1.10.0) but it is not going to be installed

python3-novaclient Depends on python3-oslo.serialization
python3-oslo.serialization Depends on python3-msgpack
python3-msgpack Depends on python3 (< 3.5)

But Debian stretch contains 3.5.3-1. So this can't be installed in mitaka/stretch. Unless we play with python3-msgpack as well:

aborrero@cloudvirt1015:~$ apt-cache policy python3-msgpack
python3-msgpack:
  Installed: (none)
  Candidate: 0.4.6-1~bpo8+1
  Version table:
     0.5.6-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     0.4.8-1 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
     0.4.6-1~bpo8+1 1001
       1001 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 Packages

aborrero@cloudvirt1015:~ $ aptitude show python3-msgpack=0.4.6-1~bpo8+1 | grep Depends
Depends: python3 (< 3.5), python3 (>= 3.4~), [...]

aborrero@cloudvirt1015:~ $ aptitude show python3-msgpack=0.4.8-1 | grep Depends
Depends: python3 (< 3.6), python3 (>= 3.5~), [...]

aborrero@cloudvirt1015:~ $ aptitude show python3-msgpack=0.5.6-1~bpo9+1 | grep Depends
Depends: python3 (< 3.6), python3 (>= 3.5~), [...]

i.e, we would need python3-msgpack from anywhere but jessie-wikimedia/openstack-mitaka-jessie

This is only taking a look at the python3-novaclient dependency chain. If we look into other packages dependency chains, the puzzle will grow bigger.

This is a cool project, but it will require me a lot of time to validate all the packages/repos in all the openstack/debian versions.
My proposal is to wait for a py3 conversion until we can reduce the amount of combos we have, i.e, wait until we are all in mitaka/stretch or even better, in newton/stretch.

Krenair changed the task status from Open to Stalled.EditedMar 21 2019, 12:55 PM
Krenair removed Krenair as the assignee of this task.
Krenair removed a project: Patch-For-Review.

Yeah., this is getting far more complicated than it should be. Let's revisit it when all the openstack servers are on stretch which should be soonish AIUI. Resolved later.

Edit: See T224561, T221769, and T221770

Xqt added a subscriber: Xqt.Jul 24 2019, 8:37 AM
Krenair changed the task status from Stalled to Open.Sep 22 2019, 11:58 AM

AFAIK no OpenStack hosts are left on Jessie anymore.