Page MenuHomePhabricator

puppet on deployment-changeprop taking forever because of systemctl start salt-minion
Closed, ResolvedPublic

Description

Not strictly a puppet error as such, but:

krenair@deployment-changeprop:~$ ps auxwf | grep -v grep | grep -A 2 puppet
root     12799  0.0  0.5  84872 11676 pts/4    S+   Aug18   0:00  |           \_ sudo puppet agent -tv
root     12800  0.3  3.7 253096 77612 pts/4    Sl+  Aug18   0:10  |               \_ /usr/bin/ruby /usr/bin/puppet agent -tv
root     13945  0.0  0.1  22488  2496 ?        Ss   Aug18   0:00  |                   \_ /bin/systemctl start salt-minion
root     28464  0.0  0.3  98148  7984 ?        Ss   00:00   0:00  \_ sshd: thcipriani [priv]
krenair@deployment-changeprop:~$ ps auxwf | grep -v grep | grep -i salt -c
339
krenair@deployment-changeprop:~$ ps auxwf | grep -v grep | grep -i salt | head -n 2
root     13945  0.0  0.1  22488  2496 ?        Ss   Aug18   0:00  |                   \_ /bin/systemctl start salt-minion
root     26285  0.0  0.0  22488  1532 ?        Ss   Aug04   0:00 /bin/systemctl start salt-minion

Event Timeline

So when I logged on yesterday, there were several hundred: systemctl start salt-minion jobs

systemd on this box seems a bit upset:

thcipriani@deployment-changeprop:~$ systemctl list-jobs
  JOB UNIT                           TYPE    STATE  
 2105 acct.service                   restart waiting
 1283 systemd-tmpfiles-clean.service start   waiting
27339 ssh.service                    stop    waiting
27563 salt-minion.service            start   waiting
 2845 changeprop.service             start   waiting
root     25025  0.0  0.0  22488   296 ?        Ss   Aug04   0:00 systemctl start ssh.service
root     23367  0.0  0.0  22488    12 ?        Ss   Aug05   0:00 /bin/systemctl start changeprop
root      6535  0.0  0.1  22488  2484 ?        Ss   05:48   0:00 /bin/systemctl start changeprop
root      7884  0.0  0.1  22488  2544 ?        Ss   06:47   0:00 /bin/systemctl start changeprop
root      9179  0.0  0.1  22488  2624 ?        Ss   07:48   0:00 /bin/systemctl start changeprop
root     10469  0.0  0.1  22488  2556 ?        Ss   08:47   0:00 /bin/systemctl start changeprop
root     11752  0.0  0.1  22488  2624 ?        Ss   09:47   0:00 /bin/systemctl start changeprop
root     13040  0.0  0.1  22488  2496 ?        Ss   10:47   0:00 /bin/systemctl start changeprop
root     14323  0.0  0.1  22488  2664 ?        Ss   11:47   0:00 /bin/systemctl start changeprop
root     15614  0.0  0.1  22488  2564 ?        Ss   12:47   0:00 /bin/systemctl start changeprop
root     16897  0.0  0.1  22488  2560 ?        Ss   13:47   0:00 /bin/systemctl start changeprop
root     18198  0.0  0.1  22488  2560 ?        Ss   14:47   0:00 /bin/systemctl start changeprop
root     19752  0.0  0.1  22488  2528 ?        Ss   15:49   0:00 /bin/systemctl start changeprop
root     21048  0.0  0.1  22488  2436 ?        Ss   16:48   0:00 /bin/systemctl start changeprop
root     22369  0.0  0.1  22488  2620 ?        Ss   17:47   0:00 /bin/systemctl start changeprop

and a dozen or so of:

root       643  0.0  0.0  21188  1624 ?        Ss   Aug01   0:11 /usr/sbin/cron -f
root     10404  0.0  0.0  44276  1520 ?        S    Aug05   0:00  \_ /usr/sbin/CRON -f
root     10407  0.0  0.0   4336   524 ?        Ss   Aug05   0:00  |   \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root     10408  0.0  0.0   4224   608 ?        S    Aug05   0:00  |       \_ run-parts --report /etc/cron.daily
root     10410  0.0  0.0   4336   564 ?        S    Aug05   0:00  |           \_ /bin/sh /etc/cron.daily/acct
root     10438  0.0  0.0   4336  1372 ?        S    Aug05   0:00  |               \_ /bin/sh /usr/sbin/invoke-rc.d acct restart
root     10456  0.0  0.0  22488    20 ?        S    Aug05   0:00  |                   \_ systemctl restart acct.service
thcipriani claimed this task.

Seems to be fixed:

thcipriani@deployment-changeprop:~$ sudo systemctl daemon-reexec
thcipriani@deployment-changeprop:~$ systemctl list-jobs
No jobs running.
thcipriani@deployment-changeprop:~$ ps aux | grep -i systemctl
thcipri+  2282  0.0  0.1  12728  2188 pts/4    S+   18:51   0:00 grep --color=auto -i systemctl
thcipriani@deployment-changeprop:~$ ps aux | grep -i salt-minion
thcipri+  2171  0.0  0.1  12728  2164 pts/4    R+   18:51   0:00 grep --color=auto -i salt-minion
root     31586  1.0  2.3 415188 49152 ?        Ssl  18:49   0:01 /usr/bin/python /usr/bin/salt-minion
thcipriani@deployment-changeprop:~$ sudo service salt-minion status
● salt-minion.service - The Salt Minion
   Loaded: loaded (/lib/systemd/system/salt-minion.service; enabled)
  Drop-In: /etc/systemd/system/salt-minion.service.d
           └─killmode.conf
   Active: active (running) since Fri 2016-08-19 18:49:20 UTC; 3min 2s ago
 Main PID: 31586 (salt-minion)
   CGroup: /system.slice/salt-minion.service
           └─31586 /usr/bin/python /usr/bin/salt-minion

Aug 19 18:49:20 deployment-changeprop systemd[1]: Started The Salt Minion.