Page MenuHomePhabricator

tools.templatetransclusioncheck hangs
Closed, ResolvedPublic

Description

Last entry in the error log is 2016-08-07 18:08:46: (server.c.1444) [note] sockets disabled, connection limit reached, but that's several days old.

Event Timeline

The host has a high load average (4), and several 100% CPU php-cgi processes from tools.jembot: T132880: tools.jembot PHP processes run out of memory and leave orphan php-cgi processes regularly?

Server also doesn't respond on localhost:

valhallasw@tools-webgrid-lighttpd-1403:~$ grep /var/run/lighttpd/templatetransclusioncheck -e 'port'
server.port = 57645
valhallasw@tools-webgrid-lighttpd-1403:~$ telnet localhost 57645
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /templatetransclusioncheck HTTP/1.0

Strace shows lighttpd in a wait loop:

sudo strace -f -p 23583
Process 23583 attached
epoll_wait(6, {}, 1025, 1000)           = 0
epoll_wait(6, {}, 1025, 1000)           = 0
epoll_wait(6, {}, 1025, 1000)           = 0

There are a bunch of php-cgi processes running:

52410    23583  0.0  0.0  52312  5756 ?        Ss   Jun02  11:14 /usr/sbin/lighttpd -f /var/run/lighttpd/templatetransclusioncheck -D
52410    23588  0.0  0.2 329336 20408 ?        Ss   Jun02   0:00 /usr/bin/php-cgi
52410    23589  0.0  0.2 329336 20516 ?        Ss   Jun02   0:00 /usr/bin/php-cgi
52410    23590  0.0  0.2 429520 16416 ?        S    Jun02   0:21 /usr/bin/php-cgi
52410    23591  0.0  0.1 429992 16104 ?        S    Jun02   0:28 /usr/bin/php-cgi
52410    24293  0.0  0.1 429224 14036 ?        S    Aug04   0:21 /usr/bin/php-cgi
52410    27543  0.0  0.1 429124 14272 ?        S    Aug04   0:22 /usr/bin/php-cgi

but these are not doing anything according to strace. Not sure what's happening here -- I'll restart the webservice, and that hopefully will resolve the issue.

valhallasw claimed this task.