The machines have currently 2 vCPUs each. The current PDF request rates are around 2 per second. If we assume an average rendering time of 15s (very generous) and a slightly elevated rate of 2.5 requests per seconds, we would need 19 concurrent renderings to be happening. Under the assumption that one worker can fire two chromium instances in parallel, the worker count would need to be 10.
- Increase the vCPU count to 10 on each machine
- Add (2?) more VMs per DC
- Have more workers than vCPUs
Probably the best option would be a combination of all of the above. Concretely, I suggest we add 4 more VMs (2 per DC), increase the vCPU count to 4, and have 6 workers running on each VM, bringing the total worker count to 24 per DC.