Page MenuHomePhabricator

Investigate alternatives to dedicated exec node for gifti's tools
Closed, DeclinedPublic

Description

I'm volunteering to help find an alternative solution to the array jobs :)

Threading has apparently been tried and is a terrible alternative. So that leaves us with:

  1. Multiprocessing (and normal grid jobs, in toollabs) with some sort of queueing system
  2. Multiprocessing and separate project with some sort of queuing system

So (1) is the better solution here and I think can be worked out easily enough, so that lets most of the current system stay in place and adds no additional sysadmin load to the user. (2) is a bit more extreme and while it would provide more control to gifti, would also add more responsibility / work - pretty unideal. Letting the status quo be is also an option.

Event Timeline

yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2015, 9:01 PM

Code is at ~tools.giftbot/dwl*.{tcl,sh}

valhallasw triaged this task as Medium priority.May 15 2015, 12:45 PM
valhallasw moved this task from Triage to Backlog on the Toolforge board.
valhallasw added a subscriber: valhallasw.

Copying from IRC:

gifti: yuvipanda: SUL-account: GiftBot (mainly dewiki, but also other German language projects), tools.giftbot, there is a crontab and there are 5 continuous jobs, one of them java, some of the crontab jobs are php; the cont jobs have aliases in .bashrc: gva, vm, mg, sga, gvm
yuvipanda: gifti does that include the jobs that run on the dedicated node?
gifti: yes, they are in jlocal dwl?.sh

The continuous jobs seem like the easiest to move first.

chasemp closed this task as Declined.Mar 17 2017, 1:16 PM
chasemp added a subscriber: chasemp.

Eventually this workload moves to k8s with all others but for now I'm marking this declined with https://phabricator.wikimedia.org/T156981#3077562