I'm volunteering to help find an alternative solution to the array jobs :)
Threading has apparently been tried and is a terrible alternative. So that leaves us with:
- Multiprocessing (and normal grid jobs, in toollabs) with some sort of queueing system
- Multiprocessing and separate project with some sort of queuing system
So (1) is the better solution here and I think can be worked out easily enough, so that lets most of the current system stay in place and adds no additional sysadmin load to the user. (2) is a bit more extreme and while it would provide more control to gifti, would also add more responsibility / work - pretty unideal. Letting the status quo be is also an option.