Maniphest T219981

openstack: clientpackages puppet entanglement
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Apr 3 2019, 12:08 PM

Description

This is blocking Trusty deprecation in HW because:

we need to decommission T218021: decommission: labtestcontrol2001.wikimedia.org which contains openstack databases we need to save
I think we can reallocate those databases to cloudcontrol2001-dev.wikimedia.org, but for that we need to properly image the server first T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch
we can't properly image the server in mitaka/stretch because the issue described below:

we have all over the roles for HW servers things like include profile::openstack::$deployment::clientpackages (or require instead of include).
That profile installs a bunch of python openstack packages (in openstack::clientpackages::common at modules/openstack/manifests/clientpackages/common.pp, client libs like:

'python-novaclient',
'python-glanceclient',
'python-keystoneclient',
'python-openstackclient',
'python-designateclient',
'python-neutronclient',

when this profile is added, these packages are installed from whatever repo is configured in the system, because openstack::clientpackages::common doesn't include any specific repo/pinning for any specific openstack/debian combo.

At a later point in the puppet catalog evaluation, we try to install openstack server components (keystone, glance, nova, neutron, etc) by means of each specific openstack server profile. And for them we do have proper repo/pinning configuration, by means of openstack::serverpackages::mitaka::stretch and friends.

But since the first python libs are already installed, this causes apt to conflict in the dependencies, or try to downgrade packages, which can cause either:

wrong openstack server packages are installed
apt failure causing puppet agent failure

Both cases are wrong and need to be fixed.

If stuff wasn't complex enough, it turns out that openstack::clientpackages::common is used * inside* CloudVPS (i.e, VM instances), so modifying that is really painful.
I already tried in https://gerrit.wikimedia.org/r/c/operations/puppet/+/500797 (already reverted) which resulted in the openstack-mitaka-jessie repo component being added to toolforge, creating apt resolver issues because packages in that component have higher pinning and again apt tries to downgrade stuff. The openstack-mitaka-jessie component is meant for HW servers only, Toolforge dependencies were not considered when I created that component.

I don't know yet how to fix this entanglement, but opened this phab task to brain dump.

Proposal 1:

let's use clientpackages stuff only for Cloud VPS instances. Try to clean servers from usage of the profile/class tree
if a script in server side requires any specific libs not installed by dependencies of server packages, let's create an explicit Package[whatever] resource for them (in the ::serverpackages namespace)

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
openstack: serverpackages: require apt-get update before moving on	operations/puppet	production	+13 -0
openstack: codfw1dev: drop even more usage of clientpackages	operations/puppet	production	+0 -1
openstack: codfw1dev: don't include clientpackages in cloudcontrol servers	operations/puppet	production	+0 -11

Customize query in gerrit

Related Objects

Mentioned Here: T218021: decommission: labtestcontrol2001.wikimedia.org
T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch

Event Timeline

• aborrero created this task.Apr 3 2019, 12:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 3 2019, 12:08 PM

• aborrero triaged this task as High priority.Apr 3 2019, 12:09 PM

• aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

• aborrero updated the task description. (Show Details)Apr 3 2019, 12:23 PM

Change 500946 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: don't include clientpackages in cloudcontrol servers

https://gerrit.wikimedia.org/r/500946

gerritbot added a project: Patch-For-Review.Apr 3 2019, 12:34 PM

Change 500946 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: don't include clientpackages in cloudcontrol servers

https://gerrit.wikimedia.org/r/500946

Change 500956 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: drop even more usage of clientpackages

https://gerrit.wikimedia.org/r/500956

Ok, now I have other suspicion for the weird behavior I'm seing:

Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Commonpackages::Mitaka/Apt::Repository[openstack-mitaka-jessie]/File[/etc/apt/sources.list.d/openstack-mitaka-jessie.list]/ensure) defined content as '{md5}b4ab161313c2b6853144638d9400b164'
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Commonpackages::Mitaka/Apt::Repository[openstack-mitaka-jessie]/File[/etc/apt/sources.list.d/openstack-mitaka-jessie.list]) Scheduling refresh of Exec[apt-get update]
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Serverpackages::Mitaka::Stretch/Apt::Pin[mitaka_stretch_nojessiebpo]/File[/etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref]/ensure) defined content as '{md5}f209f139d7d031220fdd926d72356527'
Apr  3 12:58:34 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Serverpackages::Mitaka::Stretch/Apt::Pin[mitaka_stretch_nojessiebpo]/File[/etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref]) Scheduling refresh of Exec[apt-get update]
Apr  3 12:58:59 cloudcontrol2001-dev puppet-agent[913]: (/Stage[main]/Openstack::Keystone::Service::Mitaka::Stretch/Package[keystone]/ensure) created

This means the repo got installed before keystone, but apt-get update wasn't run in time for apt to know about it.

Change 500956 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: drop even more usage of clientpackages

https://gerrit.wikimedia.org/r/500956

I want to make sure I understand our goal here... Am I right in these assumptions?

On prod hosts, we want the client packages to be pulled from the hiera-set openstack version
On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

If that's right, it seems like we maybe want to have two different classes that install the client packages: one that's in the existing openstack::<deployment> tree that depends on whatever apt-setting magic we already have, and one that's entirely outside of it, simple and explicitly only made for use on VMs.

That's roughly what you already tried, though?

• GTirloni unsubscribed.Apr 3 2019, 2:51 PM

In T219981#5081794, @Andrew wrote:

I want to make sure I understand our goal here... Am I right in these assumptions?

On prod hosts, we want the client packages to be pulled from the hiera-set openstack version

On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

If that's right, it seems like we maybe want to have two different classes that install the client packages: one that's in the existing openstack::<deployment> tree that depends on whatever apt-setting magic we already have, and one that's entirely outside of it, simple and explicitly only made for use on VMs.

That's roughly what you already tried, though?

Correct.

But I think we have another, more important problem. The openstack-mitaka-jessie repo gets installed in the same puppet agent run than the server packages are installed, but apt is not aware of the repo yet because we would need an intermediate apt-get update call before actually installing the packages. Not sure how to solve this.

In T219981#5081884, @aborrero wrote:

But I think we have another, more important problem. The openstack-mitaka-jessie repo gets installed in the same puppet agent run than the server packages are installed, but apt is not aware of the repo yet because we would need an intermediate apt-get update call before actually installing the packages. Not sure how to solve this.

This seems a common situation actually: https://stackoverflow.com/questions/10845864/run-apt-get-update-before-installing-other-packages-with-puppet

One of the proposed solutions that I like is:

package { whatever:
  ensure  => present,
  require  => Exec['apt-get update'],
}

I'm not sure we have this pattern in our current puppet codebase, so double checking the idea before introducing it.

In T219981#5081905, @aborrero wrote:

I'm not sure we have this pattern in our current puppet codebase, so double checking the idea before introducing it.

Yes! we have it already:

modules/confluent/manifests/kafka/common.pp: require => [ Exec['apt-get update']],

I will be trying this

Another way is:

apt::repository { 'wikimedia-php72':
          uri        => 'http://apt.wikimedia.org/wikimedia',
          dist       => "${::lsbdistcodename}-wikimedia",
          components => 'component/php72',
          notify     => Exec['apt_update_php'],
      }

   # First installs can trip without this
      exec {'apt_update_php':
          command     => '/usr/bin/apt-get update',
          refreshonly => true,
          logoutput   => true,
      }

Change 500977 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: serverpackages: require apt-get update before moving on

https://gerrit.wikimedia.org/r/500977

In T219981#5081794, @Andrew wrote:

On VMS hosts, we're happy for the client packages to be pulled from whatever openstack version is available in the default apt repos

This is probably fine right now but are we sure this is going to be safe in the future? If we get up to speed with OpenStack versions are our old clients from the default stretch/buster repositories still going to work?

Change 500977 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: serverpackages: require apt-get update before moving on