I don't know if this is related to the NFS stalls, but tools-webserver-01 runs out of memory from time to time; exim paniclog:
| 2013-08-03 03:13:23 daemon: fork of queue-runner process failed: Cannot allocate memory |
Daily anacron, Sun, 04 Aug 2013 06:30:12 +0000:
| /etc/cron.daily/apt: |
| FATAL -> Failed to fork. |
Weekly anacron, Sun, 04 Aug 2013 06:47:25 +0000:
| /etc/cron.weekly/apt-xapian-index: |
| FATAL -> Failed to fork. |
| run-parts: /etc/cron.weekly/apt-xapian-index exited with return code 100 |
Ganglia graphs (http://ganglia.wmflabs.org/latest/graph_all_periods.php?h=tools-webserver-01&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1375632437&g=mem_report&z=large&c=tools) look rather peaceful, with most of the memory only being used for buffers/cache.
But the webserver should never impede the system jobs from running, so this needs to be looked into. Setting up tools-webserver-03 is certainly an option, but may only defer the problem.
Version: unspecified
Severity: normal