@Ainali, I added a rough plan in the descriptions based on our earlier discussion. Feel free to add or comment.

I've done a first pass on all lists and generated ISCCs. There are now 90,329 entries in the journal with ISCC.

After a second run for T398697 there are now 99,981 ISCCs in the journal. I'll have to look into why there aren't more. Even with a few failures per batch there should be more than 100,000. There might be more duplicates among the batches that I've missed.

I compiled the numbers for all the batches in the table below. "Files" refer to how many files there were in the lists. This doesn't care about duplicates with other batches. "ISCCs" are how many ISCCs were successfully generated.

The biggest "loss" was in T398696: Make declarations for 25,000 CC BY-SA images where we had that issue with duplicates in the list itself.

@Ainali, do you think we should be happy with 19 off the target or do you want to make another small list so we get over the line?

With the images from this table we now have 100,015 ISCCs generated.

I'm currently running jobs to make declarations for the batches. I started them all using a YAML file. Because of resource quotas only two can be run at a time. The remaining jobs are still in the list with "Unable to start, out of quota for cpu, cpu". Hopefully this means that they'll start running once the first are done.

At the moment 27,402 images have been processed over 16 hours which averages at about 2.1 seconds per images. Plenty of files (3000+) take 12 s, looks like these are timeouts caused by server errors on the API.

There was an issue with the rate limiting that caused a wait between files even if they didn't need to send requests. This meant that running a batch that had some files already declared would take unnecessary long. I have a fix that needs testing, but the API is still not working.

Another issue I ran into was how to run the jobs on Toolforge. I want to run one job per batch, one after the other. I haven't figured out how to do this. There's a way to specify a job to wait until it's done before doing anything else, but that means you'll have a process running on the SSH session. Historically I don't have a good track record with that kind of things. I asked in the Wikitech list for help on this.

I've started the last batch now. It takes a little longer to run because it doesn't have the ISCCs prepared. Looking like the whole thing won't be done before I go home for the week. I'll check at the end of the day if we have enough to reach 100,000 total.

Enough declarations are now done in the last batch. 12,425 at the moment of writing, 101,366 across all batches. I'll let the rest of it run because why not.