The last Puppet run was at Tue Oct 31 05:49:07 UTC 2017 (6231 minutes ago).
Puppet runs only currently if I do that manually. I don't know why.
I fixed this already three times, by doing: sudo puppet agent -tv => logout => login => sudo puppet agent -tv => logout => login
After that, puppet was executed successfully. I don't get any errors during puppet agent -tv. So, if I execute this, puppet seems ok then, but puppet does not run automatically.
Mentioned in SAL (#wikimedia-cloud) [2017-11-04T23:27:59Z] <bd808> Created missing /tmp on cac.rcm.eqiad.wmflabs (T179778)
The expected config file exists for cron to run Puppet twice an hour.
#####################################################################
##### THIS FILE IS MANAGED BY PUPPET
##### as template('base/puppet.cron.erb')
######################################################################
7,37 * * * * root /usr/local/sbin/puppet-run > /dev/null 2>&1
@reboot root /usr/local/sbin/puppet-run > /dev/null 2>&1/var/lib/puppet/state/last_run_report.yaml is timestamped Oct 31 05:49.
/var/log on the host is empty (?) except for a wtmp log that starts Mon Oct 30 23:24:18 2017.The host also has no /tmp directory (?).
I ran /usr/local/sbin/puppet-run manually. Puppet does try to run but fails hard:
Sleeping 46 for random splay Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/apt.wikimedia.org_wikimedia_dists_trusty-wikimedia_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/ppa.launchpad.net_teward_swig3.0_ubuntu_dists_trusty_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_trusty-security_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty-updates_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty-backports_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/packages.gitlab.com_gitlab_gitlab-ce_ubuntu_dists_trusty_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/packages.gitlab.com_runner_gitlab-ci-multi-runner_ubuntu_dists_trusty_InReleaseW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.wikimedia.org trusty-wikimedia InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: GPG error: http://ppa.launchpad.net trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.ubuntu.com trusty-security InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://nova.clouds.archive.ubuntu.com trusty-updates InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: GPG error: http://nova.clouds.archive.ubuntu.com trusty-backports InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) W: GPG error: https://packages.gitlab.com trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?) E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory) Info: Retrieving pluginfacts Info: Retrieving plugin Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/content: --- /var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb 2016-09-12 09:13:26.797024405 +0000 +++ /root/puppet-file20171104-25713-1ae5qbl 2017-11-04 23:20:58.328116906 +0000 @@ -24,7 +24,7 @@ newfunction(:get_clusters, :type => :rvalue) do |args| all = {} # Ganglia config is the source of truth about clusters/site - cluster_config = function_hiera(['ganglia_clusters', {}]) + cluster_config = call_function(:hiera, ['ganglia_clusters', {}]) # Arguments are an hash of selectors selector ||= {} Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/content: content changed '{md5}8607dd91625a245e928b7382cb974dbc' to '{md5}ad456b9cf1f10f658fd6a854487e595a' Info: Loading facts Info: Caching catalog for cac.rcm.eqiad.wmflabs Info: Applying configuration version '1509836364' Error: Could not get latest version: Execution of '/usr/bin/apt-cache policy ldap-utils' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory) E: The package lists or status file could not be parsed or opened. Error: /Stage[main]/Ldap::Client::Openldap/Package[ldap-utils]/ensure: change from 2.4.31-1+nmu2ubuntu8.4 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy ldap-utils' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory) E: The package lists or status file could not be parsed or opened. Notice: /Stage[main]/Labstore::Traffic_shaping/Exec[apply_tc_config]/returns: executed successfully Error: Could not get latest version: Execution of '/usr/bin/apt-cache policy tzdata' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory) E: The package lists or status file could not be parsed or opened. Error: /Stage[main]/Base::Standard_packages/Package[tzdata]/ensure: change from 2016j-0ubuntu0.14.04 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy tzdata' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory) E: The package lists or status file could not be parsed or opened. Notice: Finished catalog run in 7.13 seconds
The missing /tmp is a least part of the problem here. @Luke081515 did you do some kind of radical manual rm trying to clean up the disk usage on this host?
I created the missing /tmp manually:
root@cac:/var/log# mkdir /tmp root@cac:/var/log# chmod 1777 /tmp root@cac:/var/log# ls -ald /tmp drwxrwxrwt 2 root root 4096 Nov 4 23:26 /tmp
After doing that /usr/local/sbin/puppet-run seems to be happier:
Sleeping 29 for random splay Splitting up /var/lib/apt/lists/partial/packages.gitlab.com_gitlab_gitlab-ce_ubuntu_dists_trusty_InRelease into data and signature failedW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com trusty InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?) W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/dists/trusty/InRelease W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/trusty-wikimedia/main/source/Sources 404 Not Found [IP: 208.80.154.22 80] W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/trusty-wikimedia/universe/source/Sources 404 Not Found [IP: 208.80.154.22 80] W: Some index files failed to download. They have been ignored, or old ones used instead. Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for cac.rcm.eqiad.wmflabs Info: Applying configuration version '1509837110' Notice: /Stage[main]/Labstore::Traffic_shaping/Exec[apply_tc_config]/returns: executed successfully Notice: Finished catalog run in 6.27 seconds
Mentioned in SAL (#wikimedia-cloud) [2017-11-04T23:46:05Z] <bd808> Restarted cron on cac.rcm.eqiad.wmflabs (T179778)