Page MenuHomePhabricator
Create Task
Maniphest T196495

Limit ability of a single user/tool to overwhealm job grid
Closed, DuplicatePublic
Actions

Description

In T196486: Concurrent generated jobs from a single user overloaded grid engine we found that a well intentioned user had started hundreds of parallel jobs on the grid. We added a specific configuration to keep that particular user from doing this again. It would be nice however to have a better circuit breaker to stop this from happening again easily.

The resource quota system looks like it would require us to list each user specifically. There is however the maxujobs global scheduler setting:

maxujobs

The maximum number of jobs any user may have running in a Sun Grid Engine cluster at the same time. If set to 0 (default) the users may run an arbitrary number of jobs.

We currently have this set in the grid config with a value of 1000 which coincidentally(?) is also the upper limit on jobs per queue. This means that the limit functionally will never take effect.

Related Objects
Search...

Event Timeline

bd808 created this task.Jun 5 2018, 6:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2018, 6:29 PM
zhuyifei1999 subscribed.Jun 5 2018, 6:37 PM

Would also help in the case of T195834#4251429. CC T67777: Limit number of jobs users can execute in parallel

Framawiki subscribed.Jun 8 2018, 5:09 PM
• Vvjjkkii renamed this task from Limit ability of a single user/tool to overwhealm job grid to 4kbaaaaaaa.Jul 1 2018, 1:06 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 4kbaaaaaaa to Limit ability of a single user/tool to overwhealm job grid.Jul 1 2018, 8:31 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
bd808 added a parent task: T199271: Upgrade the tools gridengine system.Jan 5 2019, 5:20 PM
bd808 closed this task as a duplicate of T67777: Limit number of jobs users can execute in parallel.Jan 5 2019, 5:39 PM
bd808 mentioned this in T67777: Limit number of jobs users can execute in parallel.Jan 5 2019, 6:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL