Many of my cron gridengine jobs have been failing for last 14 days throwing one of the following errors:
Traceback (most recent call last): File "/usr/bin/job", line 47, in <module> proc = subprocess.Popen(['/usr/bin/qstat', '-xml'], stdout=subprocess.PIPE) File "/usr/lib/python3.5/subprocess.py", line 676, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last): File "/usr/bin/job", line 21, in <module> import xml.etree.ElementTree File "<frozen importlib._bootstrap>", line 969, in _find_and_load File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 673, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 669, in exec_module File "<frozen importlib._bootstrap_external>", line 773, in get_code File "<frozen importlib._bootstrap_external>", line 484, in _compile_bytecode MemoryError
It never occured to me before. A majority of tasks (cca 6 or 7 of 10 tasks) last days failedwith one of these and even sent me emails with the stacktrace/traceback. It happened even for simple python scripts, that usually take miliseconds.