Page MenuHomePhabricator

Upload stopps regularly
Closed, ResolvedPublic

Description

Our uploading process stopps regularly after several files, which is quite frustrating, since we need to transfer about 40.000 files. We tried 3 times and it seems to stopp arbitary after uploading about 10, 70 or 100 files without any warning. When we go on where it stopped, it works again until next interruption, so i guess the problem is not caused by certain files.

You find the logs here:
https://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=gwtoolset&user=Nicolas+R%C3%BCck+%28WMDE%29&page=&year=&month=-1&tagfilter=

I hope someone can help,
regards, Nico

Event Timeline

Nicolas_Rueck_WMDE updated the task description. (Show Details)
Nicolas_Rueck_WMDE raised the priority of this task from to Needs Triage.
Nicolas_Rueck_WMDE changed Security from none to None.
Steinsplitter added a subscriber: Steinsplitter.
dan-nl added a comment.Dec 3 2014, 9:52 AM

this seems to be a job runner/job queue issue.

  • is there someone on the ops team who could investigate it?
  • is there a way to tell when the job runner is having a problem?
  • is there a way to see which job's are in the job queue?
Gilles triaged this task as High priority.Dec 3 2014, 1:47 PM
hoo added a subscriber: hoo.Dec 5 2014, 5:04 PM

Someone asked me about this:

hoo@terbium:~$ mwscript showJobs.php --wiki commonswiki --group
[...]
gwtoolsetUploadMediafileJob: 4 queued; 6926 claimed (0 active, 6926 abandoned); 0 delayed
gwtoolsetUploadMetadataJob: 7 queued; 0 claimed (0 active, 0 abandoned); 0 delayed
[...]

I can't see anything obvious in the logs regarding why these failed.

hoo added a comment.Dec 5 2014, 5:07 PM

Now I found something, not sure how relevant it is:

error=GWToolset\Jobs\UploadMediafileJob::run: <Other contributors: A media file with the identical title "File:Siegelmarke Finanzamt - Bad Freienwalde (Oder) W0204982.jpg" already exists in the wiki. It was edited or created by someone other than you.>
dan-nl added a comment.Dec 7 2014, 2:29 AM

thanks hoo,

the error you found for GWToolset\Jobs\UploadMediafileJob is something that is "normal" in the sense that the media file job ran, found an issue, reported it to the special:log and exited.

the gwtoolsetUploadMetadataJob is the job that creates more gwtoolsetUploadMediafileJobs. seeing 7 queued looks fine, but the important thing is that they get picked up. it would be good to know how often the jobs are run, so i analysed a bit of the special:log nicolas posted a link to above for 5 december 2014. at the end of the log, 13:01 5 december 2014, i don't know if there are still more metadata jobs in the queue or not and i don't know if the main batch job finished or not. i don't see anymore metadata jobs created so, it's possible that it did finish, but i'm not sure. at this point it would be good to be able to see if there are more metadata jobs waiting in the queue - is there a way to do that?

i probably need to create a way to monitor the totals below so that they are updated and a final or semi-final report is made during the run ...

( counts are approximate )

12:59 - 13:01

  • 41 media file jobs, failed
  • 0 media file jobs, succeeded
  • 0 metadata jobs

( gap of 40 minutes )

12:19 - 12:19

  • 7 media file jobs, failed
  • 0 media file jobs, succeeded
  • 0 metadata jobs

( gap of 18 minutes )

11:59 - 12:01

  • 21 media file jobs, failed
  • 2 media file jobs, succeeded
  • 0 metadata jobs

( gap of 27 minutes )

11:31 - 11:32

  • 15 media file jobs, failed
  • 10 media file jobs, succeeded
  • 0 metadata jobs

( gap of 12 minutes )

11:19 - 11:19

  • 16 media file jobs, failed
  • 0 media file jobs, succeeded
  • 0 metadata jobs

( gap of 15 minutes )

from 10:59 - 11:04

  • 95 media file jobs, succeeded
  • 16 media file jobs, failed
  • 24 metadata jobs, succeeded

After a cuple of days, the tool suddenly went on uploading the missing files.

Sounds like a lag in the job queue.

Nicolas_Rueck_WMDE closed this task as Resolved.Jan 5 2015, 11:06 AM
Nicolas_Rueck_WMDE claimed this task.

After about 10 days the file transfer just went on. Afterwards i had no more problems with further uploads. Apparently this issue seems to be solved.

bd808 moved this task from Backlog to Done on the MediaWiki-Core-Team board.Jan 5 2015, 4:13 PM
bd808 removed a project: MediaWiki-Core-Team.