In {T196486} we found that a well intentioned user had started hundreds of parallel jobs on the grid. We added a specific configuration to keep that particular user from doing this again. It would be nice however to have a better circuit breaker to stop this from happening again easily.
The [[http://gridscheduler.sourceforge.net/htmlman/htmlman5/sge_resource_quota.html|resource quota]] system looks like it would require us to list each user specifically. There is however the `maxujobs` [[http://gridscheduler.sourceforge.net/htmlman/htmlman5/sched_conf.html|global scheduler setting]]:
> maxujobs
>> The maximum number of jobs any user may have running in a Sun Grid Engine cluster at the same time. If set to 0 (default) the users may run an arbitrary number of jobs.
We currently have this set in the grid config with a value of `1000` which coincidentally(?) is also the upper limit on jobs per queue. This means that the limit functionally will never take effect.