Page MenuHomePhabricator

An error in Fireflytools
Closed, DuplicatePublic

Description

Traceback (most recent call last):
  File "/usr/bin/job", line 47, in <module>
    proc = subprocess.Popen(['/usr/bin/qstat', '-xml'], stdout=subprocess.PIPE)
  File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

I have got an email recently with this message. I would like to know how this issue can be solved. I guess the memory might have become full.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 19 2019, 2:08 PM
bd808 added a subscriber: bd808.

I would like to know how this issue can be solved.

Is this a failure of a webservice process or a job? Was it running on the grid engine job grid or in a Kubernetes container?

Broadly you need to either adjust your application so that it uses less total memory at runtime or adjust the runtime that the application is running under so that more memory is available to the process. For grid engine job submitted with jsub/jstart/qsub see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory for information on allocating more memory. Similar adjustments can be made for webservices running on the grid engine backend. For Kubernetes webservices we have a feature request open (T183436: Add memory limit configuration for Kubernetes pods), but no implementation at this time.

I have a small doubt. Is there any method by which .sh files which contain dump values can be cleared through Putty?

This bug seems similar to T222255. I don't think it has anything to do with -mem

Adithyak1997 added a comment.EditedMay 5 2019, 5:13 PM

@bd808

you need to either adjust your application so that it uses less total memory at runtime or adjust the runtime that the application is running under so that more memory is available to the process.

Please check the page https://tools.wmflabs.org/sge-status/. In that, when I searched for the jobs run by my tool, it showed large memory space {I guess}. So I think it has something to do with Kubernetes.

@Adithyak1997 Who is the sender of the email you are referring to by 'I have got an email recently with this message.'?

It's from Cron Daemon with the sender <root@tools.wmflabs.org>. I usually get error messages when something went wrong with the tool.

Then it has nothing to do with k8s. Cron doesn't do k8s.