Something (probably https://gerrit.wikimedia.org/r/c/operations/puppet/+/714975) stopped us from installing nrpe on cloud VMs by default.
That's potentially good, except basically every other puppet class expects to install nrpe plugins. Without including the top-level nrpe class we don't have a plugin directory, which means that there are a lot of failures. Here's a sample:
Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_client_bucket20220517-5991-vfrg1p.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/puppet/client_bucket.pp, line: 31) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_client_bucket20220517-5991-vfrg1p.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/puppet/client_bucket.pp, line: 31) Wrapped exception: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_client_bucket20220517-5991-vfrg1p.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Profile::Puppet::Client_bucket/File[/usr/local/lib/nagios/plugins/check_client_bucket]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_client_bucket20220517-5991-vfrg1p.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/puppet/client_bucket.pp, line: 31) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_timedatectl20220517-5991-1xf46wc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/systemd/timesyncd.pp, line: 23) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_timedatectl20220517-5991-1xf46wc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/systemd/timesyncd.pp, line: 23) Wrapped exception: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_timedatectl20220517-5991-1xf46wc.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Profile::Systemd::Timesyncd/File[/usr/local/lib/nagios/plugins/check_timedatectl]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_timedatectl20220517-5991-1xf46wc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/profile/manifests/systemd/timesyncd.pp, line: 23) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_microcode20220517-5991-ammoc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/base/manifests/kernel.pp, line: 104) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_microcode20220517-5991-ammoc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/base/manifests/kernel.pp, line: 104) Wrapped exception: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_microcode20220517-5991-ammoc.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Base::Kernel/File[/usr/local/lib/nagios/plugins/check_microcode]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_microcode20220517-5991-ammoc.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/base/manifests/kernel.pp, line: 104) Notice: The LDAP client stack for this host is: sssd/sudo Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo' Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_journal_pattern20220517-5991-8g6wcp.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/systemd/manifests/init.pp, line: 30) Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_journal_pattern20220517-5991-8g6wcp.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/systemd/manifests/init.pp, line: 30) Wrapped exception: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_journal_pattern20220517-5991-8g6wcp.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Systemd/File[/usr/local/lib/nagios/plugins/check_journal_pattern]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/local/lib/nagios/plugins/check_journal_pattern20220517-5991-8g6wcp.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/systemd/manifests/init.pp, line: 30)
This problem didn't appear until we built a new base image for cloud-vps, since the needed directory was baked into the old base image.