@Ainali is collecting candidate images on Projekt:CommonsDB registry/Pilotbilder.

The list is ready!

Good.

I looked at a few and notices that not all of them have license information in the SDC. Do you know for how many that's the case? So far I've only implemented fetching license from SDC since wikitext can be a bit messy.

Yeah, grabbing it from Wikitext is too challenging. But even for images without SDC, it may be retrievable in a structured form through the API, like this for example: https://commons.wikimedia.org/w/api.php?action=query&format=json&prop=imageinfo&titles=File%3AJohannes%20Vermeer%20-%20Het%20melkmeisje%20-%20Google%20Art%20Project.jpg&formatversion=2&iiprop=timestamp%7Cuser%7Cextmetadata

But perhaps we should add SDC to the ones missing as we go?

Or even simpler call: https://commons.wikimedia.org/w/api.php?action=query&format=json&prop=imageinfo&titles=File%3AJohannes%20Vermeer%20-%20Het%20melkmeisje%20-%20Google%20Art%20Project.jpg&formatversion=2&iiprop=extmetadata&iiextmetadatafilter=LicenseShortName%7CUsageTerms

For our own memory, due to the deadline, we decided to only use images with license information in SDC. Complex cases with more variation in the public domain rationale will be dealt with later.

There is still some work that needs to be done on the API side, so we'll miss the deadline for 100 declarations. I managed to make 20 requests that returned good responses.

Below is a table showing statistics for the requests. In total the 20 requests took 149 s, averaging at 7.45 s per file. There is a big difference between the largest file (164 MB, 32 s) and the smallest (0 MB, 4 s), so size is definitely a factor. I haven't looked into detail at what steps takes the most time, but I'd guess that downloading takes most and is most dependent on size. The only other step that could depend on size that I can think of is generating ISCC.

Extrapolating wildly and assuming that there will be no noticeable increase in time when the remaining metadata collecting is implemented it would take a little over 200 hours to make 100,000 declarations. That is about 8.5 days if you run around the clock. That's on my computer. On Toolforge the ISCC takes longer as will probably other processes. I don't how downloading compares though. With a bit of luck it's faster.

If it helps to keep the size down, we can query for smaller images. For example, this query gives us 100,000 CC0 images smaller than 1MB: https://w.wiki/EAa9

I think that depends on what we want to test. While we probably don't need all images to be big at least some of them should be so we now we can support them.

I'm also guessing that as part of some report we want to say something about performance. If so the images should to some extent represent what's on Commons.

In T393861#10834010, @Sebastian_Berlin-WMSE wrote:

I think that depends on what we want to test. While we probably don't need all images to be big at least some of them should be so we now we can support them.

I'm also guessing that as part of some report we want to say something about performance. If so the images should to some extent represent what's on Commons.

That's an excellent point.

From what I can see, Public Domain images usually don't have SDC yet. I understand it as there is not a consensus for the modeling yet. So for the short deadline of July, instead of both trying to get the community to agree and then run a bot for some ten thousand files, it might be easier to see if we can use the action API to find a few thousand of those. And then we can split the rest on CC0, CC BY and CC BY-SA and throw in about another hundred handpicked oddballs just to get a sense of how some extreme cases are handled.

I made about two thirds of the declarations because I missed a thing that resulted in incomplete request data. This is still considered done and we'll do T395776: Make 100,000 declarations next.