Looks like indeed the category recursion was the issue. The new run passed 190,000. Unfortunately I introduced an unrelated bug when I tried to clean up the code around TSA signatures 😞 This meant that despite this no new declarations were made.

I saw in pywikibot.Category that recursing categories can cause errors. It's not mentioned explicitly for pagegenerators.CategorizedPageGenerator, but may still be what causes the error under the hood. I changed recursion depth to 100 instead of infinite. This should be enough to get all files (I hope), but well under Python's recursion limit.

The second run stopped with an error after processing 187,200. This is suspiciously close to last time (187,215) which makes think it's not just a hiccup, but something reaching a limit in Pywikibot.

There were a few errors related to TSA. Not sure what the root cause is, but changed so that temporary files are used instead, in case there was an issue with files from old declarations remained.

When I use the preferences popup on a consumer wiki it doesn't save to the preferences page any more. @Viktoria_Hillerud_WMSE, does it work for you?

When I checked now 148,000 files had been processed. 23,567 declarations made. 518 errors out of which 171 were license related. I found and fixed one issue that was caused by only checking the first copyright status when there are multiple, like for File:Bruay-la-Buissière - Hôtel de Ville (02).JPG.

Ok, I changed so those also count as public domain.

I fixed the URLs to work and tested it with a small sample of the files that previously failed.

With the other changes made in the tasks above it's unclear if anything else is needed. We'll see if it comes up in later tests.

We did some testing and apparently this works as intended. The issue was that I encountered a bug that seems to involve caching somewhere. If I do a hard refresh (ctrl+shit+r) the special page shows the correct values.

There's still issues with the connection between the two preferences. If I change on the special page it's reflected in the popup, but if I change in the popup the special page stays the same. However after changing in the popup that value is actually used. I end up in a situation where it can say speech rate 200% on the special page, but the actual speech rate is 100%.

In T415752#11581091, @Ainali wrote:

I am guessing that the four first are licenses that CommonsDB do accept, but that the Wikidata statements are missing or not formatted exactly like what CommonsDB wants as values. I think I fixed the first one (Hungary).

After a quick rummage around in the logs it looks like many of the errors are because the licenses used are not among the ones CommonsDB accept. Here are the ones I found from a random 27 files:

This one is finally ready for review. It changes a bit how the selection player works since there's some overlap. In a few places I also simplified the behaviour, like closing the "focus player" when playing rather than figuring out how the buttons should work. There may be things that need improvements later on; the patch that is in now is meant to be "good enough".

In T395320#11573745, @Viktoria_Hillerud_WMSE wrote:

Furthermore, I’m still debugging why my edits from a wiki to an external wiki (e.g from localhost) fail with centralauthtoken errors. It looks like localhost:8080 isn’t recognized as an authorized user for making edits on the target wiki.

The job ran into an error and stopped. After a quick look in the logs this seems to be because requests to Commons failed multiple times in a row.

Here are the declarations for those files:

In T407346#11568972, @Viktoria_Hillerud_WMSE wrote:

Ah interesting! Though, the "Tools" tab behaves the same way. and that appears in the other Wikis, should we still skip this for now? @Sebastian_Berlin-WMSE

Another question, did you see my question above?

Here are the declarations in the registry. At least a couple weren't made because they didn't have a compatible license in SDC. Also, I had some issues with creating thumbnails so most don't have those.

Generating the ISCCs required some hacking, mostly because the my tool assumes that files are images. I also ran into some 429 responses when downloading files from Commons, which I haven't seen before.

Looks like this works well. Is there a way to test on a consumer wiki? Nothing happens which I guess is because the setting is off by default. Is there a way to turn it on?

I can play and pause, but the rest of the navigation is broken. Skipping by sentence just stops. Skipping by word causes errors like:

I did a quick test and it doesn't seem like the settings are synced. When I change speech rate on Special:Preferences it's reflected in the popup, but when I change in the popup it stays the same on Special:Preferences. Also if I uncheck "Read extra information for certain elements" it's still checked when I open the popup again.

After waiting for the job to progress past pending for two hours this time I changed to two CPUs (down from three) and it started immediately.

I started the Toolforge job yesterday, but when I checked this morning it was still pending after almost 20 hours. I restarted it now to see if it works better.

flushUtterancesByExpirationDateOnFile() seems a bit odd. Not sure when we'd want to remove only files since this could make files and database out of sync. I guess it's in case you have a new database, but still have old files left?

Thanks for sorting it out.

A list of all the files can be found in F71577390, 116,039 in total.

In T405330#11527853, @Viktoria_Hillerud_WMSE wrote:

When you say that the popup "writes to a wiki page", do you mean that the user settings are being saved to the user’s subpage, e.g User:someUser/Wikispeech_preferences, via the edit action, as the gadget does it?

We talked about this in person, but yes, that looks about right.

Yes. Ideally you can use Storage.loadUtterances() and just change the API parameters. May need to make two functions that calls loadUtterances(), one for content and one for messages.

It looks like you've recreated some of the logic for playing utterances. It would be better if you could re-use what we already have.

When I try this, changing things in the new popup writes to a wiki page, like when you use the gadget. It's out of sync with the default preferences and the preferences in the popup overwrite them, or at least some of them.

In T405330#11373566, @Viktoria_Hillerud_WMSE wrote:

If it's just that item that causing issues I think you can skip it for now. It seems to only appear on local installation. I don't think I've ever seen it on a "proper" wiki.

When we discussed this earlier (in T405326) we came to the conclusion that two primary buttons next to each other didn't look great. Therefore the stop button shouldn't be primary, making just the symbol red.

At the moment I'm getting errors because of missing pages (I think). Can you give me a run down on what's needed to make this work? We'd also need to include instructions somewhere how to use this for maintainers, so it could be a starting point for that.

I meant that as a way for people that don't already have an account (and don't know if they want one).

Would it be possible to add a captcha or some such for logged out users instead? I think that the workboard is the best way to get an overview of what's going on in a project. Even if it's a small hurdle to create an account it may deter people who'd otherwise be interested in seeing what's going on.

It may be best to move the readme to the root then. If you just want to install and maintain the extension you may not look in a directory named "dev".

For page content we don't send the actual text in 2. We send the segment hash. Don't remember if there was any reason we couldn't do that here.