Error:
Subject: Cron <prometheus@tools-sgebastion-07> /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom Date: Wed, 13 Mar 2019 08:06:01 +0000 From: Cron Daemon <root@tools.wmflabs.org> To: prometheus@tools.wmflabs.org /bin/sh: 1: Cannot fork
Reproducing it:
root@tools-sgebastion-07:~# su -s /bin/bash - prometheus -su: fork: retry: Resource temporarily unavailable -su: fork: retry: Resource temporarily unavailable -su: fork: retry: Resource temporarily unavailable -su: fork: retry: Resource temporarily unavailable -su: fork: Resource temporarily unavailable -su-4.4$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 64113 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 512 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 30 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited root@tools-sgebastion-07:~# cat /etc/security/limits.conf @wikidev hard cpu 720 * hard maxsyslogins 100 * hard nice 0 * - priority 10 * hard nofile 1024 * soft nofile 512 * - nproc 30
Resource usage:
root@tools-sgebastion-07:~# ps -eLf | grep prometheus-node-exporter | wc -l 30