In theory https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583030/ upgraded everything. In practice, it only upgraded hosts that had puppet working right then. Now that the puppet repo /requires/ puppet5.5/facter3 to compile catalogs, those hosts will be broken until puppet is manually upgraded.

@jbond or @MoritzMuehlenhoff can you suggest a path to manually upgrade those hosts without puppet upgrading the apt sources? Is there a better way than just manually creating the component?

@Andrew you can create:

component-facter3.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3

component-puppet5.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5

in /etc/apt/sources.list.d. run apt-get update and then apt-get install -f facter puppet

@Paladox, that's what I meant by 'manually creating the component'. But in any case, adding those three doesn't work, it just produces an endless list of dependency complaints.

facter : Depends: libboost-program-options1.62.0 but it is not installable
         Depends: libleatherman1.4.0 but it is not going to be installed
         Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
libfacter3.11.0 : Depends: libboost-filesystem1.62.0 but it is not installable
                  Depends: libboost-locale1.62.0 but it is not installable
                  Depends: libboost-program-options1.62.0 but it is not installable
                  Depends: libboost-regex1.62.0 but it is not installable
                  Depends: libboost-system1.62.0 but it is not installable
                  Depends: libcpp-hocon0.1.6 but it is not going to be installed
                  Depends: libleatherman1.4.0 but it is not going to be installed
                  Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
                  Depends: libyaml-cpp0.5v5 but it is not installable

Ah, that last dump was from a Jessie host.

Yeah, what Paladox wrote, on hosts which had Puppet broken before the facter3/puppet5 Hiera setting was applied, you can add this to sources.list:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/facter3
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/puppet5

or for jessie systems:

deb http://apt.wikimedia.org/wikimedia jessie-wikimedia component/facter3
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia component/puppet5

In the next days (as time permits) I'm planning to fold the facter3/puppet5 packages into the "main" section of jessie-wikimedia and stretch-wikimedia, this will make the separate components obsolete (and also simplify new setups)

In T248923#6012713, @Andrew wrote:

Now that the puppet repo /requires/ puppet5.5/facter3 to compile catalogs, those hosts will be broken until puppet is manually upgraded.

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

In T248923#6013379, @MoritzMuehlenhoff wrote:

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

I pushed a patch yesterday which used structured facts i.e. facts['networking']['interfaces'] which depends on facter 3. but otherwise i think this is correct

In T248923#6014121, @jbond wrote:

In T248923#6013379, @MoritzMuehlenhoff wrote:

BTW, nothing in puppet.git was specifically meant to only compile on puppet 5/ facter 3 yet, we don't use specific new features yet, it's probably rather a case of a bug fixed in the new versions which is broken in f2/p4.

I pushed a patch yesterday which used structured facts i.e. facts['networking']['interfaces'] which depends on facter 3. but otherwise i think this is correct

Ack, the only thing that's really left is to upgrade facter in Cloud VPS (not rolled out by unattended-upgrades as it installs new deps), given that it seems fine to use Puppet 5 / Facter 3 features going forward.

In T248923#6013104, @Andrew wrote:

@Paladox, that's what I meant by 'manually creating the component'. But in any case, adding those three doesn't work, it just produces an endless list of dependency complaints.

facter : Depends: libboost-program-options1.62.0 but it is not installable
         Depends: libleatherman1.4.0 but it is not going to be installed
         Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
libfacter3.11.0 : Depends: libboost-filesystem1.62.0 but it is not installable
                  Depends: libboost-locale1.62.0 but it is not installable
                  Depends: libboost-program-options1.62.0 but it is not installable
                  Depends: libboost-regex1.62.0 but it is not installable
                  Depends: libboost-system1.62.0 but it is not installable
                  Depends: libcpp-hocon0.1.6 but it is not going to be installed
                  Depends: libleatherman1.4.0 but it is not going to be installed
                  Depends: libstdc++6 (>= 5.2) but 4.9.2-10+deb8u1 is to be installed
                  Depends: libyaml-cpp0.5v5 but it is not installable

this worked fine for me on deployment-restbase01. that system is now updated and puppet is running althuogh it is showing the following warnings on each run

Warning: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]: Could not back up file of type directory
Notice: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Cassandra::Metrics/File[/etc/cassandra-metrics-collector]/ensure: removed
Warning: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector]: Could not back up file of type directory
Notice: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector]: Not removing directory; use 'force' to override