It wasn't puppetized either, so it didn't exist on the new bastions. I've killed it on tools-submit.

Change 203510 had a related patch set uploaded (by Yuvipanda):
tools: Do not allow jlocal in crontab

https://gerrit.wikimedia.org/r/203510

I've symlinked jsub to jlocal so that tools that reference it do not break, and am manually removing the references to the webwatcher script.

Change 203510 merged by Yuvipanda:
tools: Do not allow jlocal in crontab

https://gerrit.wikimedia.org/r/203510

A *lot* of crons seem to refer to them.

The common webwatcher one should already be handled by service manifests already, so it's safe to disable them, I think.

Alright, I had to revert that because plenty of tools on cron rely on being run from an exec host, which seems to be done like this. I'm going to go through and disable all the ones that seem ok to disable.

Alright, I've killed all the ones that could safely be killed, and hope the others don't cause tools-submit to die again...

manifests are, again, the long term solution to this, I think.

Cf. T66988 for the necessity; granted, "Tool Labs: support some non-work prefixes in crontabs" makes the addition look like an oversight.

Perhaps we can make one host available that can be a submit host as well, and people can schedule jobs on that one?

The change still seems to be in effect. That means that if I change my crontab, jlocal is prepended with jsub, which renders it unusable. Please effectively revert it.

Yes, I've reverted it, should be back to normal in next puppet run
(about 20mins?)

This needs to be fixed 'cleanly'.

Why does it exist? It just runs the 'job' locally.

https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Creating_a_crontab

If your cron entry only includes a brief script that, itself, sends any real work to the grid then you may skip that automatic invocation by prepending jlocal explicitly marking it as a local job. Any script or job invoked with jlocal should not be running more than a few seconds and use minimal resources; misuse of that feature may have severe impact on general reliability for all users and is not allowed.

i.e. the issue is people abusing jlocal, not the feature existing per se.

I am one of the people who pushed for a tool like jlocal in the first place. It cannot be removed without causing a lot of problems.

All but one of my cron's use jsub, the one that uses jlocal needs to say as is. It is a short script that starts up, emails a log file, prints a status message, and repeats that about 10 times and then dies. Total run time is less than 30 seconds, however the output is critical in that it needs emailed to me too. If for some reason there is a problem I need to be aware of it ASAP. I cannot wait until an error is reported as needed data may be lost.

with grid jobs its not possible (at least wasnt when cron was moved to -submit) to do that via grid without some really hackish workarounds. When tools forced all cron jobs to -submit and usage of the grid engine not all tasks are suitable for the grid. My daily email script is one of them. Instead of disabling the tool, flog those who misuse it.

In T95796#1200800, @yuvipanda wrote:

Perhaps we can make one host available that can be a submit host as well, and people can schedule jobs on that one?

I wasn't convinced then that "schedule bombs" were a threat to be considered as there is virtually no difference to other ways of submitting a plethora of jobs, and in the mean time (not by me) exec hosts have become submit hosts as well (except for the recent bunch of Trusty ones that you created). So the main reason for jlocal is no longer valid.

But there are others: One of the conveniences of cron is that it automatically mails you the output of your job. With jsub, you have to script that yourself. And while doing so you have to consider things like this. So removing jlocal can make doing something responsibly a lot harder than it needs to be.

And as @Betacommand hinted at, you don't stumble over jlocal. You have to look for it and read the documentation. And if that doesn't stop you, what will? Tools is not a padded cell, it's a collaborative environment where you have to respect that you share resources with others.

So, in short: I think we should keep jlocal.