There seem to be a large number of things that this single tool does, but the one that looks mostly likely to cause memory issues is the creation of annotated image thumbnails that are shared on Facebook (e.g. https://tools.wmflabs.org/jembot/ef/pub/20180305/1918-Joan-Josep%20Tharrats.png).

This bot was just now gobbling up CPU throughout the cluster.

I stopped the webservice, then ran 'sudo cumin --force --timeout 500 -o json "project:tools" "/usr/bin/pkill --signal 9 -u tools.jembot"' and then restarted the webservice.

Things seem better, for now... but we need a better fix.

Mentioned in SAL (#wikimedia-cloud) [2018-09-04T14:33:51Z] <andrewbogott> restarted webservice, purged stray processes. Details on T132880

I have added a link to the task description to check the busiest webgrid-lighttpd instance: https://graphite-labs.wikimedia.org/render/?width=648&height=396&_salt=1537559933.891&hideLegend=false&target=cactiStyle(highestAverage(tools.*webgrid-lighttpd*.cpu.total.user%2C8))&from=-1hours

There are again a bunch of blocked process, including new ones (several tools.wsexport for 30-90 days):

The new parent task (T153281: webgrid-lighttpd queues kill OOM jobs with SIGKILL leaving php-cgi processes behind) describes the general problem that leads to leaked processes on the job grid. It also includes work around instructions for clearing the orphan procs to free up CPU.

@-jem- have you ever tried moving your webservice from the grid engine to our Kubernetes cluster? I'm wondering if Kubernetes would be better at cleaning up when the memory limit kills things. We have both PHP 5.6 and PHP 7.2 available there:

• https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Switching_between_GridEngine_and_Kubernetes
• https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#PHP

Mentioned in SAL (#wikimedia-cloud) [2018-10-25T20:14:31Z] <andrewbogott> stopping/starting service in hopes of cleaning up stray processes, re: T132880

I just now killed off all jembot processes and restarted again.

I did do this: https://phabricator.wikimedia.org/T153281#4687515
Strictly speaking, it should not happen after a certain point. If continues happening after that change, maybe we'll have to create a new job killer script. However, this should be a long-term fix.

I confess that I didn't have a super strong case for killing things just now; one of the labvirts was under strain and I saw several (maybe 4-5?) jembot processes running there and ran straight for the hatchet. It would be useful to know how many procs is a normal amount.

This is really the same problem as T153281: webgrid-lighttpd queues kill OOM jobs with SIGKILL leaving php-cgi processes behind from the point of view of the job grid. It would be awesome if the tool could be updated/changed/tweaked to avoid hitting the job grid bug, but in the larger scope of things we need fix the grid or add a cleanup system. It may be reasonable to delay digging really deeply into the root cause and fix until we finish T199271: Upgrade the tools gridengine system and see if this problem persists.

So I take it the jobkill script isn't fixing this, then? It was definitely set to send a SIGKILL when anything went bad previously. Anything submitted before my change would likely still obey the old setting. I guess we just have to decide when we are convinced the new setting didn't do squat :)

Maybe fully resetting everything jembot?

If that was effectively done by @Andrew , and then @bd808 had to do a culling again after, then I'm convinced the new setting did nothing at all.

In T132880#4703575, @Bstorm wrote:

If that was effectively done by @Andrew , and then @bd808 had to do a culling again after, then I'm convinced the new setting did nothing at all.

The round of kills I did today (T153281#4703488) did not find any orphan jembot processes. It did find orphans from croptool, wsexport, iabot, and a few other tools that are known by me to leak occasionally.

Oooh! Ok. That might mean that as things resubmit and restart over time, the problem might actually be fixed.

Looking again at top8 -1months highest average for tools.*webgrid-lighttpd*.cpu.total.user

It seems that is mostly fixed?

There are a couple sgewebgrid instances with stable high CPU which might indicate stall process. But that might be due to a different reason.

I am not sure what got it fixed. Was it to send SIGTERM instead of SIGKILL?