Page MenuHomePhabricator
Create Task
Maniphest T95683

Puppet run interrupted by "puppet-agent: Caught TERM; calling stop"
Closed, ResolvedPublic
Actions

Description

A puppet run that applied "role::ci::slave::labs" was quite slow this time (over an hour instead of 20 minutes). It stopped before it was finished, with a vague Caught TERM; calling stop error. I've never seen this before. Reporting it in case it's significant.

Apr 10 11:13:45 integration-slave-precise-1013 puppet-agent[5393]: (/Stage[main]/Base::Monitoring::Host/Service[mpt-statusd]) Triggered 'refresh' from 1 events
Apr 10 11:13:55 integration-slave-precise-1013 puppet-agent[5393]: (/Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arabeyes]/ensure) ensure changed 'purged' to 'present'
..
..
Apr 10 11:18:28 integration-slave-precise-1013 puppet-agent[5393]: (/Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-scalable]/ensure) ensure changed 'purged' to 'present'
Apr 10 11:18:37 integration-slave-precise-1013 puppet-agent[5393]: (/Stage[main]/Mediawiki::Packages/Package[ploticus]/ensure) ensure changed 'purged' to 'present'
Apr 10 11:18:47 integration-slave-precise-1013 puppet-agent[5393]: (/Stage[main]/Mediawiki::Packages/Package[texlive-pictures]/ensure) ensure changed 'purged' to 'present'
Apr 10 11:18:50 integration-slave-precise-1013 puppet-agent[5393]: Caught TERM; calling stop

Possibly related:

Related Objects
Search...

Event Timeline

Krinkle created this task.Apr 10 2015, 1:13 PM
Krinkle raised the priority of this task from to Medium.
Krinkle updated the task description. (Show Details)
Krinkle added projects: Continuous-Integration-Infrastructure, Puppet.
Krinkle added subscribers: hashar, Krinkle, Aklapper.
hashar closed this task as Resolved.Apr 13 2015, 9:56 AM
hashar claimed this task.

Seems to be the cron based puppet run which has timeouts. We have a script running every 20 minutes and on reboot:

$ cat /etc/cron.d/puppet
12,32,52 * * * * root /usr/local/sbin/puppet-run
@reboot              root /usr/local/sbin/puppet-run

The scripts can be resumed as:

timeout --kill-after 60 300 apt-get update
timeout --kill-after 60 1800 puppet agent ...

So after 1800 seconds or 30 minutes, the puppet agent is sent TERM (15) and after 60 more seconds KILL (9).

The mediawiki::packages* puppet class install a lot of Debian packages which definitely takes a while to accomplish. On the next run, puppet will complete the installation properly.

Krinkle moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Apr 13 2015, 4:16 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL