Page MenuHomePhabricator

Puppet does not work automatically for cac.rcm.eqiad.wmflabs
Closed, ResolvedPublic

Description

The last Puppet run was at Tue Oct 31 05:49:07 UTC 2017 (6231 minutes ago).

Puppet runs only currently if I do that manually. I don't know why.

I fixed this already three times, by doing: sudo puppet agent -tv => logout => login => sudo puppet agent -tv => logout => login

After that, puppet was executed successfully. I don't get any errors during puppet agent -tv. So, if I execute this, puppet seems ok then, but puppet does not run automatically.

Event Timeline

Luke081515 created this task.

Mentioned in SAL (#wikimedia-cloud) [2017-11-04T23:27:59Z] <bd808> Created missing /tmp on cac.rcm.eqiad.wmflabs (T179778)

The expected config file exists for cron to run Puppet twice an hour.

/etc/cron.d/puppet
#####################################################################
##### THIS FILE IS MANAGED BY PUPPET
#####  as template('base/puppet.cron.erb')
######################################################################
7,37 * * * * root /usr/local/sbin/puppet-run > /dev/null 2>&1
@reboot              root /usr/local/sbin/puppet-run > /dev/null 2>&1

/var/lib/puppet/state/last_run_report.yaml is timestamped Oct 31 05:49.

/var/log on the host is empty (?) except for a wtmp log that starts Mon Oct 30 23:24:18 2017.The host also has no /tmp directory (?).

I ran /usr/local/sbin/puppet-run manually. Puppet does try to run but fails hard:

Sleeping 46 for random splay
Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/apt.wikimedia.org_wikimedia_dists_trusty-wikimedia_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/ppa.launchpad.net_teward_swig3.0_ubuntu_dists_trusty_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_trusty-security_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty-updates_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty-backports_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/partial/packages.gitlab.com_gitlab_gitlab-ce_ubuntu_dists_trusty_InReleaseCouldn't create tempfiles for splitting up /var/lib/apt/lists/packages.gitlab.com_runner_gitlab-ci-multi-runner_ubuntu_dists_trusty_InReleaseW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.wikimedia.org trusty-wikimedia InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)

W: GPG error: http://ppa.launchpad.net trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.ubuntu.com trusty-security InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://nova.clouds.archive.ubuntu.com trusty-updates InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)

W: GPG error: http://nova.clouds.archive.ubuntu.com trusty-backports InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)

W: GPG error: https://packages.gitlab.com trusty InRelease: Could not execute 'gpgv' to verify signature (is gpgv installed?)
E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory)
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/content:
--- /var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb 2016-09-12 09:13:26.797024405 +0000
+++ /root/puppet-file20171104-25713-1ae5qbl     2017-11-04 23:20:58.328116906 +0000
@@ -24,7 +24,7 @@
   newfunction(:get_clusters, :type => :rvalue) do |args|
     all = {}
     # Ganglia config is the source of truth about clusters/site
-    cluster_config = function_hiera(['ganglia_clusters', {}])
+    cluster_config = call_function(:hiera, ['ganglia_clusters', {}])

     # Arguments are an hash of selectors
     selector ||= {}

Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/content: content changed '{md5}8607dd91625a245e928b7382cb974dbc' to '{md5}ad456b9cf1f10f658fd6a854487e595a'
Info: Loading facts
Info: Caching catalog for cac.rcm.eqiad.wmflabs
Info: Applying configuration version '1509836364'
Error: Could not get latest version: Execution of '/usr/bin/apt-cache policy ldap-utils' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory)
E: The package lists or status file could not be parsed or opened.
Error: /Stage[main]/Ldap::Client::Openldap/Package[ldap-utils]/ensure: change from 2.4.31-1+nmu2ubuntu8.4 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy ldap-utils' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory)
E: The package lists or status file could not be parsed or opened.
Notice: /Stage[main]/Labstore::Traffic_shaping/Exec[apply_tc_config]/returns: executed successfully
Error: Could not get latest version: Execution of '/usr/bin/apt-cache policy tzdata' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory)
E: The package lists or status file could not be parsed or opened.
Error: /Stage[main]/Base::Standard_packages/Package[tzdata]/ensure: change from 2016j-0ubuntu0.14.04 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy tzdata' returned 100: E: Couldn't create temporary file to work with /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_Release - mkstemp (2: No such file or directory)
E: The package lists or status file could not be parsed or opened.
Notice: Finished catalog run in 7.13 seconds

The missing /tmp is a least part of the problem here. @Luke081515 did you do some kind of radical manual rm trying to clean up the disk usage on this host?

I created the missing /tmp manually:

root@cac:/var/log# mkdir /tmp
root@cac:/var/log# chmod 1777 /tmp
root@cac:/var/log# ls -ald /tmp
drwxrwxrwt 2 root root 4096 Nov  4 23:26 /tmp

After doing that /usr/local/sbin/puppet-run seems to be happier:

Sleeping 29 for random splay
Splitting up /var/lib/apt/lists/partial/packages.gitlab.com_gitlab_gitlab-ce_ubuntu_dists_trusty_InRelease into data and signature failedW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com trusty InRelease: Clearsigned file isn't valid, got 'NODATA' (does the network require authentication?)

W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/dists/trusty/InRelease

W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/trusty-wikimedia/main/source/Sources  404  Not Found [IP: 208.80.154.22 80]

W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/trusty-wikimedia/universe/source/Sources  404  Not Found [IP: 208.80.154.22 80]

W: Some index files failed to download. They have been ignored, or old ones used instead.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for cac.rcm.eqiad.wmflabs
Info: Applying configuration version '1509837110'
Notice: /Stage[main]/Labstore::Traffic_shaping/Exec[apply_tc_config]/returns: executed successfully
Notice: Finished catalog run in 6.27 seconds

Mentioned in SAL (#wikimedia-cloud) [2017-11-04T23:46:05Z] <bd808> Restarted cron on cac.rcm.eqiad.wmflabs (T179778)

bd808 claimed this task.

Puppet ran automatically from cron at Nov 5 00:08