Page MenuHomePhabricator

Request increased quota for deltabot Toolforge tool
Closed, ResolvedPublic

Description

Tool Name: deltabot
Quota increase requested: +50 count/cronjobs.batch; new limit should be "100 count/cronjob.batch"
Reason: I am currently reorganizing operation of User:DeltaBot at Wikidata, and I am running into the "50 count/cronjobs.batch" limitation (currently 44/50 used, but I have more jobs to deploy). In order to be on the safe side, I hereby request +50 cronjob.batch resources (i.e. a new limit of 100 cronjob.batch is sufficient for the foreseeable future).
Background: The bot was previously using ~10 cronjobs (distributed to the deltabot and pltools tool accounts) which ran shell scripts, and each of these shell scripts themselves started a couple of Python scripts (i.e. the shell scripts batched several independent jobs). This setup was established by the previous maintainer of the bot years ago, but it was not ideal:

  • scripts were localized in a highly disorganized way (tool accounts, folders, and cronjobs); version control only partially available
  • excessive resource quotas (memory in particular) since the most demanding Python script in a shell batch determined requested resources; most cronjobs of the old configuration requested 4Gi of memory; this often clogged the execution pipeline since parallel execution was impossible for many jobs.
  • logging was messy to a degree that it was barely useful

There was no relationship between the batched jobs, except for the same execution interval. For better maintainability, it is preferable to run one Python script per cronjob instead of batching them in shell scripts. Fortunately most jobs in the new configuration run with minimal memory demand (less than 100Mi), so that the new setup is much more considerate with memory requests—at the expense of the cronjob count.

Event Timeline

dcaro changed the task status from Open to In Progress.Oct 12 2023, 8:13 AM
dcaro moved this task from To refine to Doing on the User-dcaro board.

Done, sorry for the delay, let me know if you have any issues with it :)

dcaro moved this task from Doing to Done on the User-dcaro board.