@Hansmuller : Just to clarify - How many images are you wanting to upload, what timeframe do you want to do it in, and what rate would you like to have?

"at least every 5 minutes"

I know it says that somewhere in commons docs. That's mostly a lie and the reality of when gwtoolset gets called is significantly more complicated.

So how fast gwtoolset does things seems to be based on:

• GWToolset\Config::$metadata_job_delay (currently 1 minute)
• $wgJobBackoffThrottling['gwtoolsetUploadMetadataJob'] (currently 5 / 3600 ). I suspect this is the limiting factor
• GWToolset\Config::$mediafile_job_throttle_max (currently 20)

I think its safe to temporarily increase these limits (within reason). A major reason for them I think is to prevent saturating other people's bandwidth.

I think setting $wgJobBackoffThrottling['gwtoolsetUploadMetadataJob'] to 10/3600 would be reasonable. (However, I'd like other's opinions)

Just as a note, @Hansmuller is correct for gwtoolset being all over the place. For example, on sept 10, there was 1600 UploadMetadataJob's processed, which seems to be much more than should have been. At current settings, there should be about 5 metadata jobs an hour, 120 in a day (= ~2400 total images a day), See P2119

@Bawolff: Thanks for your interest.

• How many images? Naturalis has offered 8 million images, but of course not all are of encylopedic value. I have metadata at hand for 240.000 images, but some large collections still have to be explored.
• Timeframe: before December 1st, so 60 days to go.
• Rate: i think i can't reasonably ask for more than 2500 images/day. What would be reasonable? As a rule Naturalis images are about 300 kB.
• Bandwith: I have not experienced delays when using Uploadwizard during quite heavy uploading sessions by other users.

Best regards, Hansmuller

2500 images/day? This rate is what can be achieved by not uploading through GWToolset, but by running a script (guesstimation), which i can't. Of course, i can't do all these images mentioned above before December 1, but i should upload a fair share of various collections, which cannot be done at the present upload rate of GWToolset. So i would be very grateful for improvements. Regards, hansmuller

My GWToolset job halted tonight from 04:24 - 07:40 5 October. Sometimes it helps to submit a tiny job (5 records throttle 1), at 07:57. Repeating this procedure at around 11:15 with a 2-records job, throttle=1, after more than an hours halt did not work. (Perhaps on the contrary.) Regards, hansmu

@aaron Do you think increasing the $wgJobBackoffThrottling['gwtoolsetUploadMetadataJob'] sounds reasonable

It would be nice to have logging to know when throttling is being applied vs some other bottleneck.