Request of server-side-upload of iptc-metadata-added versions of existing files
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	C.Suthorn
	May 12 2021, 10:34 PM

Description

During the last weeks I uploaded new versions of the files I had uploaded over the last 4 1/2 years. The image data is unchanged, but I added IPTC-metadata to the files:
author
copyright
documentname
unique object name
country, city, keywords
web statement

the web statement is to be used by Google's image search to display the license of an image.

The version upload worked fine for 99.5% of the files, but for two weeks the upload of the still missing version oploads failed (mostly in the pubishing stage).

Therefore I requst a sever-side-upload of this versions of my files. I have put this images in 5 zip-archives:

http://up-and-download.de/wiki-largefiles/archive.zip
http://up-and-download.de/wiki-largefiles/archive2.zip
http://up-and-download.de/wiki-largefiles/archive3.zip
http://up-and-download.de/wiki-largefiles/archiv4.zip
http://up-and-download.de/wiki-largefiles/archiv5.zip

and added a (tab separated) csv-file with the columns filename, curid of the medium, pagename sans umlaute, uploadcomment that I would have used with my own uploads:

http://up-and-download.de/wiki-largefiles/archiv.csv

Related Objects

Mentioned In: T292954: [epic] large file uploads to commons
Mentioned Here: T284970: Video2commons scrubbing metadata
T286106: provide WebStatement and LicensorURL to users of WMF projects, with respect to Google using this MetaData in its image search to tag license information and in the advent of the EU Digital Copyright Directive in August 2021
T285981: /data/scratch NFS space too high
P16104 (An Untitled Masterwork)

Event Timeline

C.Suthorn created this task.May 12 2021, 10:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2021, 10:34 PM

• Urbanecm claimed this task.May 13 2021, 2:28 PM

Restricted Application added a project: User-Urbanecm. · View Herald TranscriptMay 13 2021, 2:28 PM

Hello, thanks for your request @C.Suthorn. Unfortunately, the data in the provided format are hard to process, because the information in archive.csv is not always up to date (the filename sometimes doesn't match).

The prefered data format would be tar.gz archives (instead of ZIP archives) with files named as they are on commons. So, instead of naming a file "51360946.tif", you would call it "Phaeno red gray night wolfsburg.tif". If you wish to update descriptions, you can put them in UTF-8 encoded txt files named "file.ext.txt" (so, for "Phaeno red gray night wolfsburg.tif", the description filename would be "Phaeno red gray night wolfsburg.tif.txt").

Archive in this format can be directly processed, and it will speed up the process.

Also, I noticed that some of the images were already re-uploaded – it would be ideal if you could remove them from the archives, to speed up the process and save some bandwidth.

Let me know if anything in my comment doesn't make sense to you.

Best,

Martin Urbanec

• Urbanecm triaged this task as Low priority.May 13 2021, 5:20 PM

• Urbanecm moved this task from Backlog to Analysis / under discussion on the Wikimedia-Site-requests board.May 13 2021, 5:36 PM

Could you give an example of a re-uploaded file? I assume that there may be some files that have been changed in the past, but still do not contain the (full) IPTC information. I would appreciate an example. tar.gz: I used up the online space that is available and uploading took long, deleting the zips and do tar.gz would be very time consuming. I can lool over the CSV file, but the majority of the files do not contain special charcters and these filename should work right away,

In T282755#7086024, @C.Suthorn wrote:

Could you give an example of a re-uploaded file? I assume that there may be some files that have been changed in the past, but still do not contain the (full) IPTC information. I would appreciate an example. tar.gz: I used up the online space that is available and uploading took long, deleting the zips and do tar.gz would be very time consuming. I can lool over the CSV file, but the majority of the files do not contain special charcters and these filename should work right away,

https://commons.wikimedia.org/wiki/File:Designer_outlets_wolfsburg_night.tif has a revision saying "adding IPTC information, Unique Object Name CS-mlldwmotvfg curid 51360952 license cc-by-sa-4.0, C.Suthorn, Designer outlets wolfsburg night.tif 83179362 2581 27", while it is in the archive CSV.

For the mismatch, see 51853993 as an example. It is listed as "Hannover - Herrenhuser Grten - Großer Garten 49.tif", but actually titled "Hannover - Herrenhäuser Gärten - Großer Garten 49.tif". That's not an issue that can be detected – if I submit that sort of input to the script, it will simply create a duplicate file – which is not what we want at all :-). It also doesn't appear to be related to special characters, rather, there is

I understand that repacking the files can be time consuming, however, it needs to happen, becuase directories structured in a different way cannot be processed by the script at all. I would've noted that before you uploaded it all, unfortunately, the ticket was created only after the initial uploading was done :-(. I won't decline this task if you won't work on that now, but I'm afraid I won't have capacity to do that myself anytime soon.

If you have shell access to the server(s) where the files are stored, it should be possible to do it all server-side (unzipping the files to a temporary directory, renaming the files to the name they should have on commons, and repacking into a tar.gz), without downloading it to your local computer first.

the revision is from 25 april, it is missing the web statement.

I do not have shell access, but I can delete the zip-archives and upload all the files again as tar.gz (1 file per tar.gz archive = "X" tar.gz files for "X" files). if the upload fails i loose only the time it takes to upload a file. also it takes to much time to create a big archive, then again, if it is to large.

the problem with pagenames is that they get changed by wikiusers.

I have added umlauts to the csv file
and I am uploading
http://up-and-download.de/wiki-largefiles/100000.tar.gz files at the moment http://up-and-download.de/wiki-largefiles/100067.tar.gz

deleted archive3 and archiv4 to make space. uploading 100141.tar.gz

deleted archi5.zip uploading 100301.tar.gz

deleted archive.zip uploading 10385.tar.gz

archive2.zip deleted uploading 100529.tar.gz

100709.tar.gz uploaded, three more to go

all files are available now

In T282755#7088296, @C.Suthorn wrote:

deleted archive.zip uploading 10385.tar.gz

urbanecm@tools-sgebastion-07  ~/tmp
$ wget http://up-and-download.de/wiki-largefiles/10385.tar.gz
--2021-05-19 18:04:07--  http://up-and-download.de/wiki-largefiles/10385.tar.gz
Resolving up-and-download.de (up-and-download.de)... 178.254.11.62
Connecting to up-and-download.de (up-and-download.de)|178.254.11.62|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-05-19 18:04:07 ERROR 404: Not Found.

urbanecm@tools-sgebastion-07  ~/tmp
$

All other filenames mentioned here did download successfully: URL list is at P16104. But there is HUGE difference in size – the zip archives had mostly over 60 GBs compressed, while this is ~5 GB in total (compressed).

urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$ du -sh
4.9G    .
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$

You are aware, that it is 712 files and I only missed a zero in one of the filenames of the milestones? The sum of the 712 tar.gz should be very similar to the sum of the 5 original zip files

complete file list:

404s:

Downloaded to scratch volume at Toolforge:

urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$ du -sh
228G    .
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$

@C.Suthorn Does this look like a plausible total size (of all files provided)?

So, the jobs I submitted to unpack the archives and move stuff to different folders by file type just finished. Unfortunately, I found two files with duplicate names in the files you provided:

urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$ cat ~/md5sum-T282755.out
cc49e8d12909562f73680318e8efa5fa  unprocessable-files/02/93019361.webm.2.webm
557c9cd786f535bf0627b4f8719f73e1  unprocessable-files/02/92635859.webm.2.webm
757374954dcaefa7a73c3e9dc9a3a460  unprocessable-files/01/92635859.webm.2.webm
455421b7f58455cf87e55653e540bad6  unprocessable-files/01/93019361.webm.2.webm
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755
$

Files in the "unprocessable-files" folder come from different files from your URL list above (or the plain webm files). I'm unable to tell you which one come from which source -- I assumed the filenames are unique across all your files, and files that weren't possible to be triaged under this assumption ended up in the unprocessable-files folder.

The files have different md5 hashes, so those are actually four different files, named using two page ID. I need advice on how to proceed with them. For now, I'll leave them where they are, and ignore them.

As only two files are affected by this: Leave this files out, once you have uploaded the other files I can run a script, that checks for discrepancies between my local files and the files online. With only two dicrepancies I can then reupload two files with Rillke's big chunked upload javascript.

Please notify me, when the uploads are done.

In T282755#7183810, @C.Suthorn wrote:

As only two files are affected by this:

Only two files that shared IDs -- I have no way to check if the file ID is correct, besides looking for conflicts.

Leave this files out, once you have uploaded the other files I can run a script, that checks for discrepancies between my local files and the files online. With only two dicrepancies I can then reupload two files with Rillke's big chunked upload javascript.

Please notify me, when the uploads are done.

Ack.

Mentioned in SAL (#wikimedia-operations) [2021-06-29T17:59:30Z] <urbanecm> Start server-side upload of ~2.5G of JPG files (T282755)

I did the following to prepare the files for a SSU (mainly, to rename them from page IDs to file names). I started with the JPG files (as they're not that big).

urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$ jsub -N urbanecm-sync-job -sync y cp -r ../processed/jpg/ .
Your job 4612104 ("urbanecm-sync-job") has been submitted
Job 4612104 exited with exit code 0.
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$ cat process-folder.sh
#!/bin/bash

function get_page_title() {
        WIKI='commonswiki'
        PAGE_ID=$1
        if [ -z "$PAGE_ID" ]; then
                echo "ERROR: Page ID not provided" >&2
                return 1
        fi

        QUERY="select page_title from page where page_namespace=6 and page_id=$PAGE_ID;"
        page_title=$(echo "$QUERY" | sql $WIKI 2>/dev/null | sed 1d)
        if [ -z "$page_title" ]; then
                echo "ERROR: No page title found for $PAGE_ID" >&2
                return 2
        fi
}

FOLDER=$1
if [ -z "$FOLDER" ]; then
        echo "ERROR: Folder not provided" >&2
        exit 1
fi

cd $FOLDER
for file in *; do
        id=$(echo $file | sed 's/\..*$//g')

        # try to acquire page id
        if ! get_page_title $id; then
                continue
        fi

        mv $file "$page_title"
done
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$ ./process-folder.sh jpg
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$ jsub -N urbanecm-compress-prepared tar czf jpg.tar.gz jpg
Your job 4612335 ("urbanecm-compress-prepared") has been submitted
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$ jsub -N md5sum -sync y md5sum jpg.tar.gz && cat ~/md5sum.out
Your job 4612452 ("md5sum") has been submitted
Job 4612452 exited with exit code 0.
2101d3d6c81ebaa2df35b3d26dfc1f4e  jpg.tar.gz
urbanecm@tools-sgebastion-07  /data/scratch/urbanecm-T282755/downloaded-data/prepared-for-ssu
$

Then, I transfered the tar.gz archive to the production cluster, and started the server-side upload. About half way through, will report via stashbot once it's done, and proceed with another batch.

jpgs done.

@C.Suthorn If you can use your scripts to check only on *.jpg files, that'd be appreciated. The rest one will take a while (especially webm files, as they need to be done more slowly).

tifs done. i'll work on webms later – as i said, they'll take much more time than the images.

@Urbanecm are you done with the space this is using on /data/scratch? The usage is a bit high after I had to do a long rsync without deletions, and I'm looking for things to clean up here T285981: /data/scratch NFS space too high

In T282755#7191258, @Bstorm wrote:

@Urbanecm are you done with the space this is using on /data/scratch? The usage is a bit high after I had to do a long rsync without deletions, and I'm looking for things to clean up here T285981: /data/scratch NFS space too high

Not yet. I still need the files stored for this task in the scratch volume. I estimate I will need them for a few more weeks. I'll make sure to delete them when I'm done with this task :-).

I thought I already wrote, that the jpgs and tifs are ok and can be deleted from scratch.

About the webms: Some of the files are large (4GiB) a significant amount is 'small' (200MB or less). Maybe it is best to order the uploads by size from largest to smallest or vice versa from smallest to largest?

Excellent, thanks a lot @C.Suthorn. I uploaded a few videos, see https://commons.wikimedia.org/w/index.php?title=Special:Log&logid=313377376, but the new metadata doesn't show anywhere on the page. Are you sure it works properly for video files?

For the images, the data was shown under "Metadata" header.

Please check if the reupload still makes sense -- note because of the bigger size and because of retranscoding, it's more expensive than images.

I have checked the 13 server-uploaded videos: the metadata is fine.

About the metadata not being shown for videos on the file description page, see T286106 and T284970

You can still see the metadata if you download the full size video and use "ffmpeg -i video.webm" on it (ffmpeg will only show the first ~80 bytes of each metadata field, if you export the metadata a complete printout is generated by ffmpeg)

Only the videos with the curids

74160702,74305338,75622881,75762675,76320869,
76604256,76822922,77651537,79066814,79262554,80015487,82427597,82895325,
83356907,83739265,83741205,84406251,84420489 ,84420539,84420562,84505710,
84696096,84735599,84735603,84787937 ,84787940,84790317,84790326,84798280, 84809612,84841545,84923003,84923593,84923595,85015693,85026363,85026373,
85043207,85043217,85043226,85043239,85043249,85043260 ,85092579 ,85153961,
86032429,8785976,91303877,91487256,91684325, 91684333,91693760 ,91849584,
91918042,91918106 ,91918184 ,91924057 ,91924197,91953165,91953196, 92168271,
92306453 ,92435328 , 92435331, 92435519 ,92435526 ,92435533 ,92435626, 92435638, 92435677, 92454278, 92454312, 92635809 ,92635859 ,92635866 ,92635904 , 93018756 ,
93018767 93018875 ,93018890 , 93018991 , 93019034 ,93019055 ,93019250 , 93019361
still need uploading. 85 files, all larger than 700MB

• Urbanecm added a project: Server-side-upload-request.Nov 16 2021, 6:42 PM

@Urbanecm: Per emails from Sep18 and Oct20 and https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup , I am resetting the assignee of this task because there has not been progress lately (please correct me if I am wrong!). Resetting the assignee avoids the impression that somebody is already working on this task. It also allows others to potentially work towards fixing this task. Please claim this task again when you plan to work on it (via Add Action... → Assign / Claim in the dropdown menu) - it would be welcome. Thanks for your understanding!

What is the state of this?

Reedy removed a project: Wikimedia-Site-requests.Mar 2 2024, 9:46 PM

No reply; assuming this is obsolete

Request of server-side-upload of iptc-metadata-added versions of existing filesClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

Request of server-side-upload of iptc-metadata-added versions of existing files
Closed, DeclinedPublic
Actions