Page MenuHomePhabricator

Please upload large files to Wikimedia Commons
Closed, ResolvedPublic

Description

Ok another try:

txts are

, , , , ,

And for videos:

wget -O "Vršič Pass.webm" https://v2c.wmflabs.org/ffacb95a02b8a615/dl.mp4.webm && wget -O "3 Kickboxing pokal Kranjske Gore 2015.webm" http://v2c.wmflabs.org/5b48a70eb89b1948/dl.mp4.webm && wget -O "Okrogla miza ob 70 letnici zaključnih bojev druge svetovne vojne na Poljani 2015.webm" http://v2c.wmflabs.org/f896aa0cd321ee1b/dl.mp4.webm && wget -O "TEVA GRC - CALCIT KAMNIK (2012-12-14).webm" http://v2c2.wmflabs.org/e90082458ec6eba3/dl.mp4.webm && wget -O "S 33-037 od Nove Gorice do Bleda 1 del, Nova Gorica - Most na Soči.webm" http://v2c2.wmflabs.org/9f96958ceefd8595/dl.mp4.webm && wget -O "GHD Lučine 2012 - dirka.webm" http://v2c2.wmflabs.org/934a6b171933198a/dl.mp4.webm

Thank you!

Event Timeline

Sporti created this task.Feb 18 2016, 2:50 PM
Restricted Application added subscribers: zhuyifei1999, JEumerus, StudiesWorld and 3 others. · View Herald TranscriptFeb 18 2016, 2:50 PM

@Sporti, can you please start batching these requests together?

Sporti renamed this task from Please upload large file to Wikimedia Commons to Please upload large files to Wikimedia Commons.Feb 18 2016, 6:39 PM
Sporti updated the task description. (Show Details)

Files are at a server of tool video2commons, so putting all in one archive like you asked on my Commons talkpage would take a lot of downloading and uploading.

@zhuyifei1999, would you be able to come up with some solution in v2c for this?

zhuyifei1999 added a comment.EditedFeb 19 2016, 12:12 AM

@zhuyifei1999, would you be able to come up with some solution in v2c for this?

How about this: I'll try to construct a link to create all of the requests in a single ticket. I'm not sure if that will exceed the max query string length.

As for zipping or tarball-ing the files, since the files are distributed on two backends (v2c.wmflabs.org (encoding01) and v2c2.wmflabs.org (encoding02), maybe more in the future), and it's currently strongly discouraged to use NFS for storage (T102240: Audit projects' use of NFS, and remove it where not necessary, T102402: Find alternative solutions for video project's use of NFS), I don't think I have a reasonably easy solution for that.

Maybe if v2c would create txt files for Commons page content at the server aswell, all you need is to produce a wget to download all the videos and txts into one place as in T127148?

Sporti updated the task description. (Show Details)Feb 19 2016, 12:32 PM

Actually, good point. @Krenair would it be fine if the tool writes all the filedescs to something like dl.mp4.webm.txt files, and have the create task link to them instead? That should prevent query string getting too long in most cases.

Actually one of the other things that would save time is to stop calling them dl.mp4.webm etc. and start using the actual upload name

So it is OK now?

Sorry, missed your description edit. Yes, the single command with all URLs and proper names is very useful (after I remember that I need to set $http_proxy and $https_proxy). A single archive (or similar command) for descriptions would be helpful, but this time I'll do it

Krenair claimed this task.Feb 19 2016, 11:43 PM

Oh, there's some very helpful encoding:

krenair@terbium:~/large-file-uploads$ ls -lh
total 14G
-rw-rw-r-- 1 krenair wikidev 4.1G Feb 18 10:07 3 Kickboxing pokal Kranjske Gore 2015.webm
-rw-rw-r-- 1 krenair wikidev  514 Feb 19 23:35 3_Kickboxing_pokal_Kranjske_Gore_2015.webm.txt
-rw-rw-r-- 1 krenair wikidev  552 Feb 19 23:34 GHD_Lu?%8Dine_2012_-_dirka.webm.txt
-rw-rw-r-- 1 krenair wikidev 966M Feb 18 17:36 GHD Lučine 2012 - dirka.webm
-rw-rw-r-- 1 krenair wikidev  538 Feb 19 23:35 Okrogla_miza_ob_70_letnici_zaklju?%8Dnih_bojev_druge_svetovne_vojne_na_Poljani_2015.webm.txt
-rw-rw-r-- 1 krenair wikidev 1.6G Feb 18 10:36 Okrogla miza ob 70 letnici zaključnih bojev druge svetovne vojne na Poljani 2015.webm
-rw-rw-r-- 1 krenair wikidev  573 Feb 19 23:34 S_33-037_od_Nove_Gorice_do_Bleda_1_del,_Nova_Gorica_-_Most_na_So?%8Di.webm.txt
-rw-rw-r-- 1 krenair wikidev 1.1G Feb 18 12:04 S 33-037 od Nove Gorice do Bleda 1 del, Nova Gorica - Most na Soči.webm
-rw-rw-r-- 1 krenair wikidev 3.6G Feb 18 09:15 TEVA GRC - CALCIT KAMNIK (2012-12-14).webm
-rw-rw-r-- 1 krenair wikidev  538 Feb 19 23:35 TEVA_GRC_-_CALCIT_KAMNIK_(2012-12-14).webm.txt
-rw-rw-r-- 1 krenair wikidev  417 Feb 19 23:35 Vrši?%8D_Pass.webm.txt
-rw-rw-r-- 1 krenair wikidev 1.8G Feb 17 08:24 Vršič Pass.webm

And I think that 4.1G file is going to be too big, sorry

Okay, this looks better:

krenair@terbium:~/large-file-uploads$ ls -lh
total 14G
-rw-rw-r-- 1 krenair wikidev 4.1G Feb 18 10:07 3 Kickboxing pokal Kranjske Gore 2015.webm
-rw-rw-r-- 1 krenair wikidev  514 Feb 19 23:35 3 Kickboxing pokal Kranjske Gore 2015.webm.txt
-rw-rw-r-- 1 krenair wikidev 966M Feb 18 17:36 GHD Lučine 2012 - dirka.webm
-rw-rw-r-- 1 krenair wikidev  552 Feb 19 23:34 GHD Lučine 2012 - dirka.webm.txt
-rw-rw-r-- 1 krenair wikidev 1.6G Feb 18 10:36 Okrogla miza ob 70 letnici zaključnih bojev druge svetovne vojne na Poljani 2015.webm
-rw-rw-r-- 1 krenair wikidev  538 Feb 19 23:35 Okrogla miza ob 70 letnici zaključnih bojev druge svetovne vojne na Poljani 2015.webm.txt
-rw-rw-r-- 1 krenair wikidev 1.1G Feb 18 12:04 S 33-037 od Nove Gorice do Bleda 1 del, Nova Gorica - Most na Soči.webm
-rw-rw-r-- 1 krenair wikidev  573 Feb 19 23:34 S 33-037 od Nove Gorice do Bleda 1 del, Nova Gorica - Most na Soči.webm.txt
-rw-rw-r-- 1 krenair wikidev 3.6G Feb 18 09:15 TEVA GRC - CALCIT KAMNIK (2012-12-14).webm
-rw-rw-r-- 1 krenair wikidev  538 Feb 19 23:35 TEVA GRC - CALCIT KAMNIK (2012-12-14).webm.txt
-rw-rw-r-- 1 krenair wikidev 1.8G Feb 17 08:24 Vršič Pass.webm
-rw-rw-r-- 1 krenair wikidev  417 Feb 19 23:35 Vršič Pass.webm.txt
krenair@terbium:~/large-file-uploads$ mwscript importImages.php commonswiki --user Sporti --comment-ext txt .
Import Images

Importing S 33-037 od Nove Gorice do Bleda 1 del, Nova Gorica - Most na Soči.webm...done.
Importing 3 Kickboxing pokal Kranjske Gore 2015.webm...failed. (* Could not write file "mwstore://local-swift-eqiad/local-public/2/2a/3_Kickboxing_pokal_Kranjske_Gore_2015.webm" because it is larger than {{PLURAL:4294967296|one byte|4294967296 bytes}}.
* Could not store file "./3 Kickboxing pokal Kranjske Gore 2015.webm" at "mwstore://local-swift-eqiad/local-public/2/2a/3_Kickboxing_pokal_Kranjske_Gore_2015.webm".
)
Importing GHD Lučine 2012 - dirka.webm...done.
Importing Okrogla miza ob 70 letnici zaključnih bojev druge svetovne vojne na Poljani 2015.webm...done.
Importing TEVA GRC - CALCIT KAMNIK (2012-12-14).webm...done.
Importing Vršič Pass.webm...done.

Found: 6
Added: 5
Failed: 1
Krenair closed this task as Resolved.Feb 20 2016, 12:07 AM