this seems to be a job runner/job queue issue.

• is there someone on the ops team who could investigate it?
• is there a way to tell when the job runner is having a problem?
• is there a way to see which job's are in the job queue?

Someone asked me about this:

hoo@terbium:~$ mwscript showJobs.php --wiki commonswiki --group
[...]
gwtoolsetUploadMediafileJob: 4 queued; 6926 claimed (0 active, 6926 abandoned); 0 delayed
gwtoolsetUploadMetadataJob: 7 queued; 0 claimed (0 active, 0 abandoned); 0 delayed
[...]

I can't see anything obvious in the logs regarding why these failed.

Now I found something, not sure how relevant it is:

error=GWToolset\Jobs\UploadMediafileJob::run: <Other contributors: A media file with the identical title "File:Siegelmarke Finanzamt - Bad Freienwalde (Oder) W0204982.jpg" already exists in the wiki. It was edited or created by someone other than you.>

thanks hoo,

the error you found for GWToolset\Jobs\UploadMediafileJob is something that is "normal" in the sense that the media file job ran, found an issue, reported it to the special:log and exited.

the gwtoolsetUploadMetadataJob is the job that creates more gwtoolsetUploadMediafileJobs. seeing 7 queued looks fine, but the important thing is that they get picked up. it would be good to know how often the jobs are run, so i analysed a bit of the special:log nicolas posted a link to above for 5 december 2014. at the end of the log, 13:01 5 december 2014, i don't know if there are still more metadata jobs in the queue or not and i don't know if the main batch job finished or not. i don't see anymore metadata jobs created so, it's possible that it did finish, but i'm not sure. at this point it would be good to be able to see if there are more metadata jobs waiting in the queue - is there a way to do that?

i probably need to create a way to monitor the totals below so that they are updated and a final or semi-final report is made during the run ...

( counts are approximate )

12:59 - 13:01

• 41 media file jobs, failed
• 0 media file jobs, succeeded
• 0 metadata jobs

( gap of 40 minutes )

12:19 - 12:19

• 7 media file jobs, failed
• 0 media file jobs, succeeded
• 0 metadata jobs

( gap of 18 minutes )

11:59 - 12:01

• 21 media file jobs, failed
• 2 media file jobs, succeeded
• 0 metadata jobs

( gap of 27 minutes )

11:31 - 11:32

• 15 media file jobs, failed
• 10 media file jobs, succeeded
• 0 metadata jobs

( gap of 12 minutes )

11:19 - 11:19

• 16 media file jobs, failed
• 0 media file jobs, succeeded
• 0 metadata jobs

( gap of 15 minutes )

from 10:59 - 11:04

• 95 media file jobs, succeeded
• 16 media file jobs, failed
• 24 metadata jobs, succeeded

After a cuple of days, the tool suddenly went on uploading the missing files.

Sounds like a lag in the job queue.