Page MenuHomePhabricator

sgebastion-07: /usr/local/bin/prometheus-puppet-agent-stats cannot fork
Closed, ResolvedPublic

Description

Error:

Subject: Cron <prometheus@tools-sgebastion-07> /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom
Date: Wed, 13 Mar 2019 08:06:01 +0000
From: Cron Daemon <root@tools.wmflabs.org>
To: prometheus@tools.wmflabs.org

/bin/sh: 1: Cannot fork

Reproducing it:

root@tools-sgebastion-07:~# su -s /bin/bash - prometheus 
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: Resource temporarily unavailable
-su-4.4$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64113
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

root@tools-sgebastion-07:~# cat /etc/security/limits.conf 
@wikidev        hard    cpu             720
*               hard    maxsyslogins    100
*               hard    nice            0
*               -       priority        10
*               hard    nofile          1024
*               soft    nofile          512
*               -       nproc           30

Resource usage:

root@tools-sgebastion-07:~# ps -eLf | grep prometheus-node-exporter | wc -l
30

Event Timeline

GTirloni created this task.Mar 13 2019, 8:15 AM
GTirloni triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2019, 8:15 AM

setrlimit(2) man page:

RLIMIT_NPROC
The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

Change 496118 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] toolforge: Allow prometheus to create up to 150 Linux threads

https://gerrit.wikimedia.org/r/496118

Change 496118 merged by GTirloni:
[operations/puppet@production] toolforge: Allow prometheus to create up to 150 Linux threads

https://gerrit.wikimedia.org/r/496118

root@tools-sgebastion-07:~# su -s /bin/bash - prometheus 
prometheus@tools-sgebastion-07:~$ /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom
prometheus@tools-sgebastion-07:~$ echo $?
0
GTirloni closed this task as Resolved.Mar 13 2019, 8:29 AM