When investigating T239831 I noticed that recent stretch reimages were not using Puppet 5: https://debmonitor.wikimedia.org/packages/puppet
We need to fix the pinning (or stop using a separate component for Puppet 5 and just move it to stretch-wikimedia/main, but that might cause issues with Cloud VPS)
Change 554549 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Setup apt pinning for puppet 5 / facter 3 on stretch/jessie
Change 554549 merged by Muehlenhoff:
[operations/puppet@production] Setup apt pinning for puppet 5 / facter 3 on stretch/jessie
Mentioned in SAL (#wikimedia-operations) [2019-12-12T09:46:42Z] <moritzm> upgrading recently reimaged stretch hosts back to puppet 5 / facter 3 T239832
Change 559509 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Readd the late-install hack until WMCS switched to Puppet 5 / Facter 3 as well
Change 559509 merged by Muehlenhoff:
[operations/puppet@production] Readd the late-install hack until WMCS switched to Puppet 5 / Facter 3 as well
Mentioned in SAL (#wikimedia-operations) [2020-01-02T11:42:18Z] <moritzm> reimaging mw2277 to validate fix for puppet5/facter3 installation on new installs T239832
Mentioned in SAL (#wikimedia-operations) [2020-01-02T12:32:18Z] <moritzm> upgrade recently reimaged hosts to facter 3 T239832
Mentioned in SAL (#wikimedia-operations) [2020-01-02T12:41:27Z] <moritzm> upgrade recently reimaged hosts to puppet 5 T239832
New Stretch/Jessie installations are now fixed by reintroducing the late-install.sh hack. I also upgraded all recently reimaged hosts so that production is again consistently on Puppet 5/ Facter 3 (with the odd exception of labtestpuppetmaster2001)