In T376979#12022292, @Zache wrote:

Just FYI about QLever syncing. In april we tried to use QLever as backend for Ukbot (bot which calculates points for editing competitions) and least for wikidata sync, it misses some of the wikidata item creations (ukbot github) and probably edits too. It was not syncing lag problem as there were newer wikidata item creations which could be found on Qlever so it seemed just to be skipping some edits. I suppose that same feature is with SDC sync too.

Ok, so looks like timeline breaks when using swedish capital letter 'Å' in name: bar:Fredrik text:"Fredrik Åkesson".
If you mangle the name into using another character timeline works but name is wrong.
Workaround is to use HTML-encoded character like Å instead of correct character.

Making any edit on the timeline results in error : https://fi.wikipedia.org/wiki/Arch_Enemy

Here's one error message received upon launching bot-script:

articles have misused name-attribute in ref-tags: some have used single-quotes and others entirely without quotes. these articles should be fixed by bot run to use proper duoble quotes which are the standard way.

According to T323849 and https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits, pywikibot does not yet have OAuth 2 and owner-only tokens, which would be needed for correct handling of rate limit? (T407987).

This seems to be a bug in pywikibot, especially considering the rate limits at: https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits

some label in location maps have shifted places to outside the location map, these work in preview but not after saving in article pages.
see if there is a simple fix (calculation order?) for the affected pages before larger updates.

versionhistory coloring does not work in article pages although it works in preview and template documentation page.

some infoboxes have sub-template for player outfit colors, this has used a pattern that does not work after parsoid changes (outfit appear beside that actual infobox).

old-style escaping of table start/end: template documentation pages have been made separate in the past and the escaping is remnant of past.

I am guessing this is the right ticket to report an issue. In fi-wiki there is now broken rendering in the case where another template is within infobox (for basketball player outfit), example: [[:fi:Suomen koripallomaajoukkue]]. Also template documentation pages are not using table markup correctly, example: [[fi:Malline:Kirjaviite/ohje]].

..and of course python breaks if you move virtualenvs anywhere..

another seems to be if external IP address changes due to something ISP does, which is unavoidable since people don't have static IP these days..

it seems pywikibot starts sleeping indefinitely when changing from one (python) virtualenv to another with same user for running different helper script (backfill hashes). the second script might not be needed if we would download every image and calculate hashes before uploading, but with copy-upload method we have to do that afterwards.

for siiri, download via redirect is added and handling of id with siiri urn is added, needs testing and debugging

made changes, wikidata-site session is now a singleton instance and reused.

also looking up already uploaded images we need to consider that Commons might have encoded id and data from Finna has unencoded version so lookup must unencode appropriately

reopening since it is not that simple after all.

added

there's another case for the encoding: structured data in commons needs the same extra encoding as the view, template already has it. also finna-id in structured data seems to work but claims where the url is set as source needs the same encoding as well.

closing for now unless new cases appear

close for now unless something else comes up

added url encoding for cases where view needs it while keeping id as-is for looking up the original, SLS and FMP urls affected in the view for now.

to expand on this: currently values like institution, type of work, subject and author are stored separately in the data model. to make logic based on combinations needs code outside of the model itself. for instance, if a physical object stores author in creditline and manufacturer of object in non-present author that kind of combination is not supported in the model since they are separate entities. likewise if certain institution uses "manufacturer" for a photographer's role again those are stored separately and additional code is needed.
code itself isn't the issue, but having the same logic applied to template(s), structured data and/categories and filename means spreading out the code in multiple places, which is a problem for maintainng and adding new such conditions.

other way would be to continue spreading case-by-case logic in the code itself..

added a quick fix, does not fix everything and should use different method

copy-upload ei ole sallittu domainista: kuvat.fmp.fi

classification part has information such as "50 vesi- ja ilmaliikenne" - they might be useful for categories or overly generic to be of use

inscriptions (things written on physical object) and related works (where image has been used, printed, shown, exhibited..) have been added to uploader. these come with new uploads, the helper script has been used to backfill these previously.

some images have empty set in "subjectPlaces" but they have location under events/valmistus/places. in some cases this is a simple string but some add another part of structured elements there.

In T418198#11661572, @Atieno wrote:

The pywikibot query from my end returns the hundreds of results. @Ipr1 are the 2 users you are using in the same usergroup? The official usergroups are listed here User rights and groups

example case: name of institution is stored on import, but that is not changed afterwards. only potential wikidata id is checked, not if the institution itself has changed.

images from old sources might have low quality with compression artifacts and hashing method does not seem reliable with the usual difference allowed. uploading a new version might be a better idea in any case.

note that manufacturing information seems to mainly exists for non-photographic items.

added parsing of information for materials and methods of creating (events/valmistus, type = "valmistus"), and adding that to the template metadata fields. script hasn't been run after modifications yet but tested slightly.

additionally photographers that needed categories and/or wikidata items were discovered and other categories that were necessary were added.

script was run and missing information was added. additionally there were found images that needed manual intevention (previously unknown obsolete domain, faulty id numbers, entirely missing source links or just badly formatted urls).

these are now (at least those that were found to be incorrect) converted to record-format urls from collection-format. update was done manually to affected images (102): there is no way to guess the correct number by script unless you do tedious guessing from the plaintext free-form description and that would only apply to the sls-collections anyway. so it would be of little use and a lot of effort to do it by script, and hopefully there won't be more of these.

tiffinfo gives warnings from EXIF metadata, so it is not the actual image data that is causing the problem upload?

See related tasks T358943 and T352942 about pillow bugging on image conversion (resulting in empty image) which also prevents using it for imagehashes in some cases.

modified and ran script for these images as well

There is also another old domain helsingforsbilder.fi with same issue.

Should find a better way to determine which are needed/missing based on what we can find. As far as I know, there is no public API for photography museum's information so it is manual work to find out what is needed in Wikidata.
Also, need to find other sources since that is incomplete and we would need information for many photographers who have images available in Finna.

The motivation for the search with bot is to find and update links to current Finna-service with correct identifiers as the old domain as changed and the backend has changed. Similar update has been done before for kuvakokoelmat.fi -urls which no longer works but images are found in Finna-service. Additionally, bot must compare images to confirm the links and identifiers are correct (see User:FinnaUploader page in Commons).

Quarry does not give any results while Commons external link search gives upto 500 results: https://commons.wikimedia.org/w/index.php?title=Special:LinkSearch&limit=500&offset=0&target=https%3A%2F%2Fhelsinkikuvia.fi

Problem is likely somewhere on the server side but there is no way to know where.

Tämä on osittaisena tehty isoimmille paikkakunnille. Lisäkäsittelyä tarvittaisiin jos on tietylle rakennukselle (kartanolle tms.) oma luokka tai vastaavaa.

Script is working in several cases but some unexpected cases might not be correctly handled: these are usually cases where normal freeform-system is not followed correctly in any case.
Running the script still requires monitoring and checking results.