Page MenuHomePhabricator

Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia
Closed, ResolvedPublic

Description

Nodepool (>0.1.1) is going to require the Python module os-client-config >= 1.2.0. 1.3.0-1 has landed in Debian experimental. https://packages.qa.debian.org/p/python-os-client-config.html

Dependencies are straightforward and fulfilled for Jessie

Source: https://packages.debian.org/source/experimental/python-os-client-config
Binary: https://packages.debian.org/experimental/python-os-client-config

The source package generates three binary packages:

  • python-os-client-config
  • python-os-client-config-doc
  • python3-os-client-config

We only need the first one (python2) but can probably upload all three of them on our apt.wikimedia.org.

(similar to T102880: Backport python-diskimage-builder 0.1.46 from testing to jessie-wikimedia )

the package report an invalid watch file format pointing to pypi. Nothing to worry about imho:

debian-watch-file-unsupported-pypi-url
line 2

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 7 2015, 9:54 AM
hashar updated the task description. (Show Details)Jul 7 2015, 9:57 AM
hashar set Security to None.
hashar updated the task description. (Show Details)

git clone git://anonscm.debian.org/openstack/python-os-client-config.git

dch --bpo

Modify changelog entry from jessie-backports to jessie-wikimedia

Build with:

GIT_PBUILDER_AUTOCONF=no DIST=jessie WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder

Fails with:

pbuilder-satisfydepends-dummy : 
Depends: python-keystoneclient (>= 1:1.1.0) but it is not going to be installed.
Depends: python-oslosphinx (>= 2.5.0) but it is not going to be installed.
Depends: python-oslotest (>= 1.5.1) but it is not going to be installed.
Depends: python3-oslotest (>= 1.5.1) but it is not going to be installed.

Available versions:

PackageJessieJessie backports
python-keystoneclient0.10.11.3.0
python-oslotest1.0.01.5.1
python3-oslotest1.0.01.5.1

Looking at the source requirements.txt, it only has PyYAML >= 3.1.0. The other dependencies are solely to run tests.

I am not sure how to inject backports in the cowbuilder image for jessie-wikimedia.

Andrew added a subscriber: Andrew.Aug 3 2015, 7:40 PM

I'm a bit confused by this task. Is it still just that you need python-os-client-config downloaded from sid and imported into the jessie-wikimedia repo? Or do we need to spin out other tasks for the associated packages?

Andrew triaged this task as Medium priority.Aug 7 2015, 4:40 PM

The package has build dependencies which are not matched in Jessie (see table above). So when backporting the sid package to jessie, we can remove from debian/control the dependencies related to tests and build the package with DEB_BUILD_OPTIONS=nocheck.

The binary requirements are solely python-pbr and python-yaml. So potentially it is just about:

diff --git a/debian/control b/debian/control
index 5d1e239..7d268cf 100644
--- a/debian/control
+++ b/debian/control
@@ -13,25 +13,11 @@ Build-Depends: debhelper (>= 9),
                python3-all,
                python3-pbr,
                python3-setuptools,
-Build-Depends-Indep: python-coverage,
-                     python-extras,
-                     python-fixtures,
-                     python-hacking,
-                     python-keystoneclient (>= 1:1.1.0),
+Build-Depends-Indep: python-extras,
                      python-oslosphinx (>= 2.5.0),
-                     python-oslotest (>= 1.5.1),
-                     python-testscenarios,
-                     python-testtools,
                      python-yaml,
-                     python3-fixtures,
 #                     python3-keystoneclient (>= 1:1.1.0),
-                     python3-oslotest (>= 1.5.1),
-                     python3-subunit,
-                     python3-testscenarios,
-                     python3-testtools,
                      python3-yaml,
-                     subunit,
-                     testrepository,
 Standards-Version: 3.9.6
 Vcs-Browser: http://anonscm.debian.org/gitweb/?p=openstack/python-os-client-config.git
 Vcs-Git: git://anonscm.debian.org/openstack/python-os-client-config.git

If we cared about tests, we would need a way to inject in the build env the backports repository. But I don't think it matters.

hashar added a comment.EditedSep 3 2015, 9:22 AM

I'm a bit confused by this task. Is it still just that you need python-os-client-config downloaded from sid and imported into the jessie-wikimedia repo? Or do we need to spin out other tasks for the associated packages?

Nodepool is at version 0.1.1, the commit after introduces a dependency upon python-shade (T107267) which depends on python-os-client-config hence this task.

From my earlier comment (T104967#1491403) the package building in jessie-wikimedia fails to match some dependencies:

pbuilder-satisfydepends-dummy : 
Depends: python-keystoneclient (>= 1:1.1.0) but it is not going to be installed.
Depends: python-oslosphinx (>= 2.5.0) but it is not going to be installed.
Depends: python-oslotest (>= 1.5.1) but it is not going to be installed.
Depends: python3-oslotest (>= 1.5.1) but it is not going to be installed.

But they are available in jessie-backports:

PackageJessieJessie backports
python-keystoneclient0.10.11.3.0
python-oslotest1.0.01.5.1
python3-oslotest1.0.01.5.1

Our Jessie production servers no more includes jessie-backports, @Andrew published some in our jessie-wikimedia/thirdparty component.

Where I am hitting a wall is attempting to build that python-os-client-config from sid using the our package_builder puppet module. Since it does not grab material from either jessie-backports or jessie-wikimedia/thirdparty, the dependencies are missing.

Maybe the building hook can be adjusted to inject backports/thirdparty in the cowbuilder image ( modules/package_builder/templates/D01apt.wikimedia.org.erb).

@fgiunchedi shoutout as a packaging guru :D

Can you provide any guidance?

I believe package_builder will try importing from jessie-wikimedia if WIKIMEDIA=yes is passed on the command line, that should work @hashar !

WIKIMEDIA=yes does not include either jessie-backports or jessie-wikimedia/thirdparty.

We would need our magic wrapper / hooks to recognize some other env variables to include Jessie backports and a way to list additional components (i.e.: thirdparty).

My only attempt was with:

GIT_PBUILDER_AUTOCONF=no DIST=jessie WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder

I haven't dig in the code to figure out how we could inject thirdparty, there is probably no way to do it with our current hook.

We would probably need to publish the dependencies in thirdparty which is the component used on our machine (jessie-backports is not included).


Side note: maybe we can consider using scap to deploy a virtualenv. That would nicely solve the dependency madness for Zuul and Nodepool.

afaict our puppet hooks for jessie does include thirdparty

package_builder::pbuilder_hook { 'jessie':
    distribution => 'jessie',
    components   => 'main backports thirdparty',
    basepath     => $basepath,
}
copper:~$ cat /var/cache/pbuilder/hooks/jessie/D01apt.wikimedia.org 
#!/bin/sh

# Avoid running hooks if wikimedia is not specified
if [ "${WIKIMEDIA}" = "yes" ]; then
	cat > /etc/apt/sources.list.d/wikimedia.list <<-'EOF'
	deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty
	deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty
EOF
	cat > /etc/apt/preferences.d/wikimedia.pref <<-'EOF'
	Package: *
	Pin: release o=Wikimedia
	Pin-Priority: 1001
EOF
	apt-get install wget -y
	wget -O - -o /dev/null http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg | apt-key add -
	apt-get update
fi

afaict our puppet hooks for jessie does include thirdparty

package_builder::pbuilder_hook { 'jessie':
    distribution => 'jessie',
    components   => 'main backports thirdparty',
    basepath     => $basepath,
}

seems to resolve this?

Thanks for all the attention. The pbuider hook defines jessie-wikimedia/backports but the packages are in upstream jessie-backports/main which is not included.

So it seems we cherry pick from upstream backport repo to our backports component.

From T104967#1601353:

The packages are available in upstream jessie-backports:

PackageJessieJessie backports
python-keystoneclient0.10.11.3.0
python-oslotest1.0.01.5.1
python3-oslotest1.0.01.5.1

Since I have filled the bug python-keystoneclient has been cherry picked from upstream to our jessie-wikimedia/backports probably while we upgraded OpenStack:

$ apt-cache policy python-keystoneclient
python-keystoneclient:
  Installed: 1:1.3.0-2~bpo8+1
  Candidate: 1:1.3.0-2~bpo8+1
  Version table:
 *** 1:1.3.0-2~bpo8+1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
        100 /var/lib/dpkg/status
     1:0.10.1-2+deb8u1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages

So I guess now we need to import from jessie-backports/main to jessie-wikimedia/backports:

  • python-oslosphinx
  • python-oslotest
  • python3-oslotest

The modules are used to run tests and generate documentation, so it should be harmless for our OpenStack structure.

ok I've uploaded oslo-sphinx python-oslotest and openstack-pkg-tools (build-dep for oslo-sphinx) to jessie-wikimedia

jessie-wikimedia|backports|source: oslo-sphinx 2.5.0-1~bpo8+1
jessie-wikimedia|backports|amd64: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|i386: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|source: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|amd64: openstack-pkg-tools 24~bpo8+1
jessie-wikimedia|backports|i386: openstack-pkg-tools 24~bpo8+1
jessie-wikimedia|backports|source: openstack-pkg-tools 24~bpo8+1

Nice, seems that fixed it :-} I backported python-os-client-config_1.6.0-4 using:

git clone git://anonscm.debian.org/openstack/python-os-client-config.git
dch --bpo
# Bump version to 1.6.0-4~bpo8+1
DEB_BUILD_OPTIONS=nocheck git-buildpackage --git-ignore-new  --git-dist=jessie-wikimedia --git-pbuilder --git-debian-branch=HEAD

It managed to grab the missing deps from jessie-wikimedia/backports. End result ready for upload is at either one of:

:)

Once done, we can backport python-shade (T107267)

fgiunchedi closed this task as Resolved.Nov 5 2015, 4:13 PM
fgiunchedi claimed this task.

ok I've uploaded python-os-client-config

root@carbon:~# reprepro -C backports  include jessie-wikimedia ~filippo/deb/python-os-client-config_1.6.0-4~bpo8+1_amd64.changes 
Exporting indices...