Page MenuHomePhabricator

Trusty puppet 4 approach
Closed, ResolvedPublic

Description

How do we want to handle puppet agent packages for our ubuntu trusty servers (~80 hosts)?

I see 3 options:

  1. Use the puppetlabs trusty puppet-agent package. Version 1.8.3-1trusty provides puppet 4.8.2. This package bundles many different pieces of software including facter-3.5.1 and openssl-1.0.2j (full list below).
Contains the following components:
augeas 1.4.0
cpp-hocon 0.1.4
cpp-pcp-client 1.3.1
curl 7.51.0
dmidecode 2.12
facter 3.5.1
hiera 3.2.2
leatherman 0.10.1
libxml2 2.9.4
libxslt 1.1.29
marionette-collective 2.9.1
openssl 1.0.2j
puppet 4.8.2
puppet-ca-bundle 1.0.7
pxp-agent 1.3.2
ruby-2.1.9 2.1.9
ruby-augeas 0.5.0
ruby-shadow 2.3.3
ruby-stomp 1.3.3
rubygem-deep-merge 1.0.1
rubygem-fast_gettext 1.1.0
rubygem-gettext 3.2.2
rubygem-gettext-setup 0.6
rubygem-hocon 0.9.3
rubygem-locale 2.1.2-2
rubygem-net-ssh 2.9.2
rubygem-semantic_puppet 0.1.2
rubygem-text 1.3.1
runtime
shellpath 2015-09-18
virt-what 1.14

Also, the puppetlabs puppet-agent package uses a different layout on disk from the current package (/etc/puppetlabs, /opt/puppetlabs, etc.) and will require modification of our trusty puppet agent puppetization to work.

  1. Rebuild the jessie puppet 4 packages for trusty. Probably a cleaner approach, but also means we take on the maintenance of these packages ourselves.
  1. Leave it alone! Our puppet masters support 3.x agents through the rack middleware and the number of trusty hosts should decrease over time.

Event Timeline

herron triaged this task as Normal priority.Dec 14 2017, 4:09 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2017, 4:09 PM
faidon added a subscriber: faidon.Dec 14 2017, 4:14 PM

Option (1) is not really an option -- as I said in the other task, those packages are horrible, a bigger maintenance burden, and they are also very different from everything else we use. Between (2) and (3) I think option (2) is the way to go here. This is what we have done in the past, and past experience shows that the extra maintenance burden isn't much. Note that at some point we'll have to take on maintenance for even jessie/stretch systems, to go with e.g. Puppet 5.

Andrew added a subscriber: Andrew.Dec 15 2017, 2:58 PM

(update: @herron is in the process of backporting)

A first stab at Trusty packages for puppet 4.8.2 and dependencies (hiera, ruby-deep-merge) have been built on boron (in /var/cache/pbuilder/result/trusty-amd64/) and appear to be working after cursory testing (it installs and puppet runs) on puppet-4-trusty-[12].puppet.eqiad.wmflabs.

Testing so for has been using a local file apt source, and the packages throw a "cannot be authenticated" warning this way. Will something need to be done to sign these correctly beyond adding them to a proper repo with reprepro?

Also is there a testing repo somewhere that could be used for further validation?

A first stab at Trusty packages for puppet 4.8.2 and dependencies (hiera, ruby-deep-merge) have been built on boron (in /var/cache/pbuilder/result/trusty-amd64/) and appear to be working after cursory testing (it installs and puppet runs) on puppet-4-trusty-[12].puppet.eqiad.wmflabs.

All your packages are no-change backports, so the builds seems fine (trusty and jessie are fairly close, so not surprising).

Bu you should rebuild with modified version numbers, it's confusing to see +bpo versions on trusty (and reprepro will probably stumble over the fact that different binaries with the same version are tracked).

You can extract the source package using "dpkg-source -x foo.dsc", then run "dch" to generate a new changelog entry (I suggest you simply append ~trusty1 to the original Debian versions, e.g. 1.3.1-1~trusty1). Changelog entry can be something like "No change backport for trusty" (or if you needed to make actual backport changes, please list them). This is useful since we might need to rebuild puppet in the future (we did this for security updates in the past).

Set the distribution line to "trusty-wikimedia".

Testing so for has been using a local file apt source, and the packages throw a "cannot be authenticated" warning this way. Will something need to be done to sign these correctly beyond adding them to a proper repo with reprepro?

We don't use signed uploads in our infrastructure (yet). You can copy the built files (source, debs and .changes file) to install1002.wikimedia.org. Then run

reprepro -C main include trusty-wikimedia foo.changes

Also is there a testing repo somewhere that could be used for further validation?

We have the experimental component (somewhat deprecated, but still exists), but for your purpose I suggest you copy the files to one of the ocg* hosts. They no longer run OCG, but they're still installed with trusty, so you can safely install the new packages with "dpkg -i foo.deb" and test general puppet operation before uploading to apt.wikimedia.org

Thanks @MoritzMuehlenhoff!

New packages have been built following this guidance, uploaded to apt.wikimedia.org and trusty hosts have been upgraded to the puppet 4 agent today.

Once T184017 is resolved I think we'll be done here!

herron closed this task as Resolved.Jan 9 2018, 4:30 PM
herron claimed this task.

This week @chasemp discovered an issue where puppet apply breaks on trusty hosts with Error: Evaluation Error: Error while evaluating a Function Call, uninitialized constant RGen::ECore::ELong. We traced this back the older version of ruby-rgen (dependency of the puppet package) that is available on trusty.

Upgrading to the jessie ruby-rgen package solved this issue, so ruby-rgen-0.7.0-1 has been backported from jessie to trusty and uploaded to apt.wikimedia.org.

Mentioned in SAL (#wikimedia-operations) [2018-01-18T22:36:50Z] <herron> added ruby-rgen-0.7.0-1 (backported package from jessie) to trusty-wikimedia apt repo (T182894)