ThumbnailRenderJob occasionally fails with 429 errors as seen in logstash
The absolute maximum concurrency for the job execution is 20, which could probably be lowered to 10, but that's on the edge on not being able to clear up any backlog that might occur.
I'm not particularly sure where the 429s come from exactly, but my understanding is that they originate from too many requests for thumbnails coming from a single job runner host, and since all the job runners are now behind an LVS we have no control over which hosts will the jobs end up on, so any concurrency higher then the thumbor worker number of 4 is not safe.
Is my assumption correct that it's the pool counter throttling is the reason for 429s? Do we need pool counter throttling for the job queue requests? Should we try decreasing concurrency and looking what happens, or remove/increase the throttling for job queue requests?