Page MenuHomePhabricator
Create Task
Maniphest T221451

An error in Fireflytools
Closed, DuplicatePublic
Actions

Description

Traceback (most recent call last):
  File "/usr/bin/job", line 47, in <module>
    proc = subprocess.Popen(['/usr/bin/qstat', '-xml'], stdout=subprocess.PIPE)
  File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

I have got an email recently with this message. I would like to know how this issue can be solved. I guess the memory might have become full.

Related Objects

Event Timeline

Adithyak1997 created this task.Apr 19 2019, 2:08 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptApr 19 2019, 2:08 PM
bd808 added a project: Tools.Apr 21 2019, 9:44 PM
bd808 subscribed.

I would like to know how this issue can be solved.

Is this a failure of a webservice process or a job? Was it running on the grid engine job grid or in a Kubernetes container?

Broadly you need to either adjust your application so that it uses less total memory at runtime or adjust the runtime that the application is running under so that more memory is available to the process. For grid engine job submitted with jsub/jstart/qsub see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory for information on allocating more memory. Similar adjustments can be made for webservices running on the grid engine backend. For Kubernetes webservices we have a feature request open (T183436: Add memory limit configuration for Kubernetes pods), but no implementation at this time.

Adithyak1997 added a comment.May 3 2019, 6:56 PM

I have a small doubt. Is there any method by which .sh files which contain dump values can be cleared through Putty?

zhuyifei1999 subscribed.May 5 2019, 4:42 PM

This bug seems similar to T222255. I don't think it has anything to do with -mem

Adithyak1997 added a comment.EditedMay 5 2019, 5:13 PM

@bd808

In T221451#5127795, @bd808 wrote:
you need to either adjust your application so that it uses less total memory at runtime or adjust the runtime that the application is running under so that more memory is available to the process.

Please check the page https://tools.wmflabs.org/sge-status/. In that, when I searched for the jobs run by my tool, it showed large memory space {I guess}. So I think it has something to do with Kubernetes.

zhuyifei1999 added a comment.May 5 2019, 5:15 PM

@Adithyak1997 Who is the sender of the email you are referring to by 'I have got an email recently with this message.'?

Adithyak1997 added a comment.May 5 2019, 5:17 PM

It's from Cron Daemon with the sender <root@tools.wmflabs.org>. I usually get error messages when something went wrong with the tool.

zhuyifei1999 closed this task as a duplicate of T222255: tools-sgecron-01 virtual memory allocation error at midnight and noon UTC.May 5 2019, 5:18 PM

Then it has nothing to do with k8s. Cron doesn't do k8s.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits