Page MenuHomePhabricator

openstack: clientpackages puppet entanglement
Closed, ResolvedPublic

Description

This is blocking Trusty deprecation in HW because:

we have all over the roles for HW servers things like include profile::openstack::$deployment::clientpackages (or require instead of include).
That profile installs a bunch of python openstack packages (in openstack::clientpackages::common at modules/openstack/manifests/clientpackages/common.pp, client libs like:

'python-novaclient',
'python-glanceclient',
'python-keystoneclient',
'python-openstackclient',
'python-designateclient',
'python-neutronclient',

when this profile is added, these packages are installed from whatever repo is configured in the system, because openstack::clientpackages::common doesn't include any specific repo/pinning for any specific openstack/debian combo.

At a later point in the puppet catalog evaluation, we try to install openstack server components (keystone, glance, nova, neutron, etc) by means of each specific openstack server profile. And for them we do have proper repo/pinning configuration, by means of openstack::serverpackages::mitaka::stretch and friends.

But since the first python libs are already installed, this causes apt to conflict in the dependencies, or try to downgrade packages, which can cause either:

  • wrong openstack server packages are installed
  • apt failure causing puppet agent failure

Both cases are wrong and need to be fixed.

If stuff wasn't complex enough, it turns out that openstack::clientpackages::common is used * inside* CloudVPS (i.e, VM instances), so modifying that is really painful.
I already tried in https://gerrit.wikimedia.org/r/c/operations/puppet/+/500797 (already reverted) which resulted in the openstack-mitaka-jessie repo component being added to toolforge, creating apt resolver issues because packages in that component have higher pinning and again apt tries to downgrade stuff. The openstack-mitaka-jessie component is meant for HW servers only, Toolforge dependencies were not considered when I created that component.

I don't know yet how to fix this entanglement, but opened this phab task to brain dump.

Proposal 1:

  • let's use clientpackages stuff only for Cloud VPS instances. Try to clean servers from usage of the profile/class tree
  • if a script in server side requires any specific libs not installed by dependencies of server packages, let's create an explicit Package[whatever] resource for them (in the ::serverpackages namespace)

Event Timeline

aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 500946 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: don't include clientpackages in cloudcontrol servers

https://gerrit.wikimedia.org/r/500946

Change 500946 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: don't include clientpackages in cloudcontrol servers

https://gerrit.wikimedia.org/r/500946

Change 500956 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: drop even more usage of clientpackages

https://gerrit.wikimedia.org/r/500956

Ok, now I have other suspicion for the weird behavior I'm seing:

Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Commonpackages::Mitaka/Apt::Repository[openstack-mitaka-jessie]/File[/etc/apt/sources.list.d/openstack-mitaka-jessie.list]/ensure) defined content as '{md5}b4ab161313c2b6853144638d9400b164'
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Commonpackages::Mitaka/Apt::Repository[openstack-mitaka-jessie]/File[/etc/apt/sources.list.d/openstack-mitaka-jessie.list]) Scheduling refresh of Exec[apt-get update]
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Serverpackages::Mitaka::Stretch/Apt::Pin[mitaka_stretch_nojessiebpo]/File[/etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref]/ensure) defined content as '{md5}f209f139d7d031220fdd926d72356527'
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Serverpackages::Mitaka::Stretch/Apt::Pin[mitaka_stretch_nojessiebpo]/File[/etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref]) Scheduling refresh of Exec[apt-get update]
Apr  3 12:58:59 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Keystone::Service::Mitaka::Stretch/Package[keystone]/ensure) created

This means the repo got installed before keystone, but apt-get update wasn't run in time for apt to know about it.

Change 500956 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: drop even more usage of clientpackages

https://gerrit.wikimedia.org/r/500956

I want to make sure I understand our goal here... Am I right in these assumptions?

  1. On prod hosts, we want the client packages to be pulled from the hiera-set openstack version
  2. On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

If that's right, it seems like we maybe want to have two different classes that install the client packages: one that's in the existing openstack::<deployment> tree that depends on whatever apt-setting magic we already have, and one that's entirely outside of it, simple and explicitly only made for use on VMs.

That's roughly what you already tried, though?

I want to make sure I understand our goal here... Am I right in these assumptions?

  1. On prod hosts, we want the client packages to be pulled from the hiera-set openstack version
  2. On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

If that's right, it seems like we maybe want to have two different classes that install the client packages: one that's in the existing openstack::<deployment> tree that depends on whatever apt-setting magic we already have, and one that's entirely outside of it, simple and explicitly only made for use on VMs.

That's roughly what you already tried, though?

Correct.

But I think we have another, more important problem. The openstack-mitaka-jessie repo gets installed in the same puppet agent run than the server packages are installed, but apt is not aware of the repo yet because we would need an intermediate apt-get update call before actually installing the packages. Not sure how to solve this.

But I think we have another, more important problem. The openstack-mitaka-jessie repo gets installed in the same puppet agent run than the server packages are installed, but apt is not aware of the repo yet because we would need an intermediate apt-get update call before actually installing the packages. Not sure how to solve this.

This seems a common situation actually: https://stackoverflow.com/questions/10845864/run-apt-get-update-before-installing-other-packages-with-puppet

One of the proposed solutions that I like is:

package { whatever:
  ensure  => present,
  require  => Exec['apt-get update'],
}

I'm not sure we have this pattern in our current puppet codebase, so double checking the idea before introducing it.

I'm not sure we have this pattern in our current puppet codebase, so double checking the idea before introducing it.

Yes! we have it already:

modules/confluent/manifests/kafka/common.pp: require => [ Exec['apt-get update']],

I will be trying this

Another way is:

apt::repository { 'wikimedia-php72':
          uri        => 'http://apt.wikimedia.org/wikimedia',
          dist       => "${::lsbdistcodename}-wikimedia",
          components => 'component/php72',
          notify     => Exec['apt_update_php'],
      }

   # First installs can trip without this
      exec {'apt_update_php':
          command     => '/usr/bin/apt-get update',
          refreshonly => true,
          logoutput   => true,
      }

Change 500977 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: serverpackages: require apt-get update before moving on

https://gerrit.wikimedia.org/r/500977

  1. On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

This is probably fine right now but are we sure this is going to be safe in the future? If we get up to speed with OpenStack versions are our old clients from the default stretch/buster repositories still going to work?

Change 500977 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: serverpackages: require apt-get update before moving on

https://gerrit.wikimedia.org/r/500977

aborrero claimed this task.

This seems fixed by now. No need to panic. I was able to reimage the affected server today with much better results.