Page MenuHomePhabricator

Puppet failures with tlsproxy::envoy on cloud-vps
Closed, ResolvedPublic

Description

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /etc/puppet/modules/profile/manifests/tlsproxy/envoy.pp, line: 72, column: 28) on node deployment-restbase01.deployment-prep.eqiad.wmflabs

My understanding is that this failure happens because that class uses a factor 3 fact that isn't present on cloud-vps. We either need to fast-track the facter upgrade on cloud-vps or not use new puppetisms until cloud-vps is upgraded.

Event Timeline

Andrew created this task.Mar 30 2020, 9:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2020, 9:15 PM

In theory https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583030/ upgraded everything. In practice, it only upgraded hosts that had puppet working right then. Now that the puppet repo /requires/ puppet5.5/facter3 to compile catalogs, those hosts will be broken until puppet is manually upgraded.

@jbond or @MoritzMuehlenhoff can you suggest a path to manually upgrade those hosts without puppet upgrading the apt sources? Is there a better way than just manually creating the component?

@Andrew you can create:

component-facter3.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3

component-puppet5.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5

in /etc/apt/sources.list.d. run apt-get update and then apt-get install -f facter puppet

@Paladox, that's what I meant by 'manually creating the component'. But in any case, adding those three doesn't work, it just produces an endless list of dependency complaints.

facter : Depends: libboost-program-options1.62.0 but it is not installable
         Depends: libleatherman1.4.0 but it is not going to be installed
         Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
libfacter3.11.0 : Depends: libboost-filesystem1.62.0 but it is not installable
                  Depends: libboost-locale1.62.0 but it is not installable
                  Depends: libboost-program-options1.62.0 but it is not installable
                  Depends: libboost-regex1.62.0 but it is not installable
                  Depends: libboost-system1.62.0 but it is not installable
                  Depends: libcpp-hocon0.1.6 but it is not going to be installed
                  Depends: libleatherman1.4.0 but it is not going to be installed
                  Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
                  Depends: libyaml-cpp0.5v5 but it is not installable

Ah, that last dump was from a Jessie host.

Yeah, what Paladox wrote, on hosts which had Puppet broken before the facter3/puppet5 Hiera setting was applied, you can add this to sources.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5

or for jessie systems:

deb http://apt.wikimedia.org/wikimedia jessie-wikimedia component/facter3
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia component/puppet5

In the next days (as time permits) I'm planning to fold the facter3/puppet5 packages into the "main" section of jessie-wikimedia and stretch-wikimedia, this will make the separate components obsolete (and also simplify new setups)

Now that the puppet repo /requires/ puppet5.5/facter3 to compile catalogs, those hosts will be broken until puppet is manually upgraded.

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

Aklapper renamed this task from Pupppet failures with tlsproxy::envoy on cloud-vps to Puppet failures with tlsproxy::envoy on cloud-vps.Mar 31 2020, 7:05 AM

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

I pushed a patch yesterday which used structured facts i.e. facts['networking']['interfaces'] which depends on facter 3. but otherwise i think this is correct

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

I pushed a patch yesterday which used structured facts i.e. facts['networking']['interfaces'] which depends on facter 3. but otherwise i think this is correct

Ack, the only thing that's really left is to upgrade facter in Cloud VPS (not rolled out by unattended-upgrades as it installs new deps), given that it seems fine to use Puppet 5 / Facter 3 features going forward.

@Paladox, that's what I meant by 'manually creating the component'. But in any case, adding those three doesn't work, it just produces an endless list of dependency complaints.

facter : Depends: libboost-program-options1.62.0 but it is not installable
         Depends: libleatherman1.4.0 but it is not going to be installed
         Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
libfacter3.11.0 : Depends: libboost-filesystem1.62.0 but it is not installable
                  Depends: libboost-locale1.62.0 but it is not installable
                  Depends: libboost-program-options1.62.0 but it is not installable
                  Depends: libboost-regex1.62.0 but it is not installable
                  Depends: libboost-system1.62.0 but it is not installable
                  Depends: libcpp-hocon0.1.6 but it is not going to be installed
                  Depends: libleatherman1.4.0 but it is not going to be installed
                  Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
                  Depends: libyaml-cpp0.5v5 but it is not installable

this worked fine for me on deployment-restbase01. that system is now updated and puppet is running althuogh it is showing the following warnings on each run

Warning: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]: Could not back up file of type directory
Notice: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]/ensure: removed
Warning: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector]: Could not back up file of type directory
Notice: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector]: Not removing directory; use 'force' to override
JHedden triaged this task as Medium priority.
JHedden closed this task as Resolved.May 5 2020, 4:50 PM