Page MenuHomePhabricator

puppet broken on integration WMCS instances due to openstack Debian packages
Closed, ResolvedPublic

Description

Since Friday March 15 around 13:00 UTC, puppet is failing on the CI WMCS instances. When puppet class Openstack::Clientpackages::Common is applied, apt installation of openstack libraries fail entirely.

A full puppet log from integration-slave-docker-1051.integration.eqiad.wmflabs is P8215

As an example, here the output for python-novaclient

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python-novaclient' returned 100: Reading package lists...


The following packages have unmet dependencies:
 python-novaclient : Depends: python-keystoneauth1 (>= 2.1.0) but it is not going to be installed
                     Depends: python-oslo.i18n (>= 2.1.0) but it is not going to be installed
                     Depends: python-oslo.serialization (>= 1.10.0) but it is not going to be installed
                     Depends: python-oslo.utils (>= 3.5.0) but it is not going to be installed
                     Depends: python-pbr (>= 1.8) but 0.8.2-1 is to be installed
E: Unable to correct problems, you have held broken packages.
Error: /Stage[main]/Openstack::Clientpackages::Common/Package[python-novaclient]/ensure: change from purged to present failed

And I have:

apt-cache policy python-novaclient
python-novaclient:
  Installed: (none)
  Candidate: 2:3.3.1-3~bpo8+1
  Package pin: 2:3.3.1-3~bpo8+1
  Version table:
     2:3.3.1-3~bpo8+1 1002
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/openstack-mitaka-jessie amd64 Packages
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
     2:3.3.1-3~bpo8+1 1002
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
     2:2.18.1-1 1002
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

Seems that is related to the introduction of jessie-wikimedia/openstack-mitaka-jessie component in apt with eg:

/etc/apt/preferences.d/jessie_mitaka_pinning_python_novaclient.pref
Package: python-novaclient
Pin: version 2:3.3.1-3~bpo8+1
Pin-Priority: 1002

Event Timeline

hashar created this task.Mar 18 2019, 11:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2019, 11:30 AM
aborrero triaged this task as High priority.Mar 18 2019, 12:46 PM
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.
aborrero added a subscriber: aborrero.

Change 497595 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] Revert "puppet_alert: Email projectadmins instead of members"

https://gerrit.wikimedia.org/r/497595

that is somehow due to ::profile::openstack::main::clientpackages. I have reverted https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495757/ / ff5d0e075eb7ae9900d89dd5e865c8f1e7e8d971 and that fixed the apt issue

Mentioned in SAL (#wikimedia-releng) [2019-03-19T19:56:45Z] <hashar> integration: sudo cumin --force '*' 'rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list' # T218559

You have a conflicting pin of the python-pbr package

modules/zuul/manifests/init.pp
if os_version('debian == jessie') {
    include ::apt
    apt::pin { 'python-pbr':
        pin      => 'release o=debian',
        priority => '1002',
        before   => Package['zuul'],
    }
}

Mentioned in SAL (#wikimedia-releng) [2019-03-19T20:23:19Z] <hashar> integration: sudo cumin --force '*' 'rm /etc/apt/preferences.d/jessie_mitaka_pinning_*' # T218559

Krenair renamed this task from puppet broken on integration WMCS instances due to cumin/openstack Debian packages to puppet broken on integration WMCS instances due to openstack Debian packages.Mar 19 2019, 8:52 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-08T12:31:21Z] <hashar> contint2001: upgraded python-pbr 0.8.2-1 -> 1.10.0-1 # T218559

Change 502207 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] zuul: stop pinning python-pbr

https://gerrit.wikimedia.org/r/502207

Change 497595 abandoned by Hashar:
Revert "puppet_alert: Email projectadmins instead of members"

Reason:
It is no more cherry picked on the integration puppet master. Zuul no more needs an apt pin preference for python-pbr which is removed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /502207/

https://gerrit.wikimedia.org/r/497595

hashar closed this task as Resolved.Apr 8 2019, 12:43 PM
hashar claimed this task.

Seems good now.

Change 502207 merged by Dzahn:
[operations/puppet@production] zuul: stop pinning python-pbr

https://gerrit.wikimedia.org/r/502207

Mentioned in SAL (#wikimedia-operations) [2019-05-07T10:16:30Z] <hashar> contint1001, contint2002: rm /etc/apt/preferences.d/python_pbr.pref /etc/apt/preferences.d/python-pbr.pref # T218559

Mentioned in SAL (#wikimedia-operations) [2019-05-07T10:16:56Z] <hashar> contint1001: upgrading python-pbr from 0.8.2-1 to 1.10.0-1 , no more needed with recent Zuul # T218559

Cleanup complete, Zuul in production now uses Distribute to get the version and I have upgrade python-pbr to 1.10.0 from jessie-wikimedia/backports.