There is a general problem I have with this plan, which is that as we stand, the API and appserver clusters are reserved (as much as possible) to live requests from the website or other services to perform their duties.

Asynchronous processing should instead hit a separate cluster - one we can isolate/turn off temporarily in the event of extreme need, and most importantly, that we don't mind overloading by mistake as it won't have an user-visible impact.

Also, on jobrunners we can keep laxer timeouts for requests, making it easier for jobs to complete even under pressure.

Thanfully there is a solution to this problem - we should make the jobrunners finally be "regular" appservers with just tweaked parameters. There was quite some work in that direction by @hnowlan which we can revive if we decide to go forward with this plan.

There's a few other minor things to figure out (for instance: where will we have a log like the one the jobqueue creates now) that can be further analyzed when the plan is more defined.