git clone git://anonscm.debian.org/openstack/python-os-client-config.git

dch --bpo

Modify changelog entry from jessie-backports to jessie-wikimedia

Build with:

GIT_PBUILDER_AUTOCONF=no DIST=jessie WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder

Fails with:

pbuilder-satisfydepends-dummy : 
Depends: python-keystoneclient (>= 1:1.1.0) but it is not going to be installed.
Depends: python-oslosphinx (>= 2.5.0) but it is not going to be installed.
Depends: python-oslotest (>= 1.5.1) but it is not going to be installed.
Depends: python3-oslotest (>= 1.5.1) but it is not going to be installed.

Available versions:

Looking at the source requirements.txt, it only has PyYAML >= 3.1.0. The other dependencies are solely to run tests.

I am not sure how to inject backports in the cowbuilder image for jessie-wikimedia.

I'm a bit confused by this task. Is it still just that you need python-os-client-config downloaded from sid and imported into the jessie-wikimedia repo? Or do we need to spin out other tasks for the associated packages?

@hashar can you clarify?

The package has build dependencies which are not matched in Jessie (see table above). So when backporting the sid package to jessie, we can remove from debian/control the dependencies related to tests and build the package with DEB_BUILD_OPTIONS=nocheck.

The binary requirements are solely python-pbr and python-yaml. So potentially it is just about:

diff --git a/debian/control b/debian/control
index 5d1e239..7d268cf 100644
--- a/debian/control
+++ b/debian/control
@@ -13,25 +13,11 @@ Build-Depends: debhelper (>= 9),
                python3-all,
                python3-pbr,
                python3-setuptools,
-Build-Depends-Indep: python-coverage,
-                     python-extras,
-                     python-fixtures,
-                     python-hacking,
-                     python-keystoneclient (>= 1:1.1.0),
+Build-Depends-Indep: python-extras,
                      python-oslosphinx (>= 2.5.0),
-                     python-oslotest (>= 1.5.1),
-                     python-testscenarios,
-                     python-testtools,
                      python-yaml,
-                     python3-fixtures,
 #                     python3-keystoneclient (>= 1:1.1.0),
-                     python3-oslotest (>= 1.5.1),
-                     python3-subunit,
-                     python3-testscenarios,
-                     python3-testtools,
                      python3-yaml,
-                     subunit,
-                     testrepository,
 Standards-Version: 3.9.6
 Vcs-Browser: http://anonscm.debian.org/gitweb/?p=openstack/python-os-client-config.git
 Vcs-Git: git://anonscm.debian.org/openstack/python-os-client-config.git

If we cared about tests, we would need a way to inject in the build env the backports repository. But I don't think it matters.

In T104967#1504200, @Andrew wrote:

I'm a bit confused by this task. Is it still just that you need python-os-client-config downloaded from sid and imported into the jessie-wikimedia repo? Or do we need to spin out other tasks for the associated packages?

Nodepool is at version 0.1.1, the commit after introduces a dependency upon python-shade (T107267) which depends on python-os-client-config hence this task.

From my earlier comment (T104967#1491403) the package building in jessie-wikimedia fails to match some dependencies:

pbuilder-satisfydepends-dummy : 
Depends: python-keystoneclient (>= 1:1.1.0) but it is not going to be installed.
Depends: python-oslosphinx (>= 2.5.0) but it is not going to be installed.
Depends: python-oslotest (>= 1.5.1) but it is not going to be installed.
Depends: python3-oslotest (>= 1.5.1) but it is not going to be installed.

But they are available in jessie-backports:

Our Jessie production servers no more includes jessie-backports, @Andrew published some in our jessie-wikimedia/thirdparty component.

Where I am hitting a wall is attempting to build that python-os-client-config from sid using the our package_builder puppet module. Since it does not grab material from either jessie-backports or jessie-wikimedia/thirdparty, the dependencies are missing.

Maybe the building hook can be adjusted to inject backports/thirdparty in the cowbuilder image ( modules/package_builder/templates/D01apt.wikimedia.org.erb).

@fgiunchedi shoutout as a packaging guru :D

Can you provide any guidance?

I believe package_builder will try importing from jessie-wikimedia if WIKIMEDIA=yes is passed on the command line, that should work @hashar !

WIKIMEDIA=yes does not include either jessie-backports or jessie-wikimedia/thirdparty.

We would need our magic wrapper / hooks to recognize some other env variables to include Jessie backports and a way to list additional components (i.e.: thirdparty).

My only attempt was with:

GIT_PBUILDER_AUTOCONF=no DIST=jessie WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder

I haven't dig in the code to figure out how we could inject thirdparty, there is probably no way to do it with our current hook.

We would probably need to publish the dependencies in thirdparty which is the component used on our machine (jessie-backports is not included).

Side note: maybe we can consider using scap to deploy a virtualenv. That would nicely solve the dependency madness for Zuul and Nodepool.

afaict our puppet hooks for jessie does include thirdparty

package_builder::pbuilder_hook { 'jessie':
    distribution => 'jessie',
    components   => 'main backports thirdparty',
    basepath     => $basepath,
}

copper:~$ cat /var/cache/pbuilder/hooks/jessie/D01apt.wikimedia.org 
#!/bin/sh

# Avoid running hooks if wikimedia is not specified
if [ "${WIKIMEDIA}" = "yes" ]; then
	cat > /etc/apt/sources.list.d/wikimedia.list <<-'EOF'
	deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty
	deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty
EOF
	cat > /etc/apt/preferences.d/wikimedia.pref <<-'EOF'
	Package: *
	Pin: release o=Wikimedia
	Pin-Priority: 1001
EOF
	apt-get install wget -y
	wget -O - -o /dev/null http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg | apt-key add -
	apt-get update
fi

In T104967#1761540, @fgiunchedi wrote:

afaict our puppet hooks for jessie does include thirdparty

package_builder::pbuilder_hook { 'jessie':
    distribution => 'jessie',
    components   => 'main backports thirdparty',
    basepath     => $basepath,
}

seems to resolve this?

Thanks for all the attention. The pbuider hook defines jessie-wikimedia/backports but the packages are in upstream jessie-backports/main which is not included.

So it seems we cherry pick from upstream backport repo to our backports component.

From T104967#1601353:

The packages are available in upstream jessie-backports:

Since I have filled the bug python-keystoneclient has been cherry picked from upstream to our jessie-wikimedia/backports probably while we upgraded OpenStack:

$ apt-cache policy python-keystoneclient
python-keystoneclient:
  Installed: 1:1.3.0-2~bpo8+1
  Candidate: 1:1.3.0-2~bpo8+1
  Version table:
 *** 1:1.3.0-2~bpo8+1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
        100 /var/lib/dpkg/status
     1:0.10.1-2+deb8u1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages

So I guess now we need to import from jessie-backports/main to jessie-wikimedia/backports:

• python-oslosphinx
• python-oslotest
• python3-oslotest

The modules are used to run tests and generate documentation, so it should be harmless for our OpenStack structure.

ok I've uploaded oslo-sphinx python-oslotest and openstack-pkg-tools (build-dep for oslo-sphinx) to jessie-wikimedia

jessie-wikimedia|backports|source: oslo-sphinx 2.5.0-1~bpo8+1
jessie-wikimedia|backports|amd64: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|i386: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|source: python-oslotest 1.5.1-1~bpo8+1
jessie-wikimedia|backports|amd64: openstack-pkg-tools 24~bpo8+1
jessie-wikimedia|backports|i386: openstack-pkg-tools 24~bpo8+1
jessie-wikimedia|backports|source: openstack-pkg-tools 24~bpo8+1

Nice, seems that fixed it :-} I backported python-os-client-config_1.6.0-4 using:

git clone git://anonscm.debian.org/openstack/python-os-client-config.git
dch --bpo
# Bump version to 1.6.0-4~bpo8+1
DEB_BUILD_OPTIONS=nocheck git-buildpackage --git-ignore-new  --git-dist=jessie-wikimedia --git-pbuilder --git-debian-branch=HEAD

It managed to grab the missing deps from jessie-wikimedia/backports. End result ready for upload is at either one of:

• https://people.wikimedia.org/~hashar/debs/python-os-client-config_1.6.0-4/
• https://people.wikimedia.org/~hashar/debs/python-os-client-config_1.6.0-4.tar.gz
• scp terbium.eqiad.wmnet:/home/hashar/public_html/debs/python-os-client-config_1.6.0-4/ .

:)

Once done, we can backport python-shade (T107267)