In T131553#6647417, @mat_kelcey wrote:

e,g, how you generated "https://tools.wmflabs.org/multichill/queries2/commons/paintings_without_wikidata_in_painting_category.txt"

In T268645#6645735, @Xqt wrote:

Looks like it is the same as T192912

Due to

if code == 'failed-save' and \
   self.action == 'wbeditentity' and \
   self._is_wikibase_error_retryable(result['error']):
    self.wait()
    continue

it seems that action is not 'wbeditentity'; any idea what it was?

In T257938#6645626, @Cparle wrote:

@AnneT the config patch has been merged so you're good to go on the UI patch. Options are

• haslicense:cc-by
• haslicense:cc-by-sa
• haslicense:unrestricted
• haslicense:other

Hi folks, happy to see activity on this task. Current code at https://github.com/multichill/toollabs/blob/master/bot/wikidata/find_painting_images.py#L704

@Cparle @matthiasmullie I noticed the change was merged. Do you have a pointer where the actual mappings are stored? See https://github.com/multichill/toollabs/blob/master/bot/commons/own_work_sdoc.py#L66 for a list of mappings that I would expect (minus the variants).

Thanks for adding screen. Can you add /usr/bin/mysql to the whitelist too? Just the client part. Mariadb server will auto disconnect any long open sessions and the client will just reconnect when a session is needed again.

I just noticed this also breaks https://commons.wikimedia.org/wiki/Special:MediaSearch if you sort it by "recency".

I looked around in old bugs and found T129046 . I think it went like this:

• Several Wikibase page_props were added for some unknown reason (you would have to look in the code where these are actually used now)
• I wanted to track items without statements (T129037) and easiest was to use the page_props (T129046)
• Some other page_props were added to the RDF later

Very confusing indeed. I updated https://www.wikidata.org/wiki/MediaWiki:Protect-text

Still means the wording of the protect page is incorrect. Currently it says "all users" and "allow only administrators". The "all users" is not correct because changing the protection to that won't make it possible for all users to edit, only for autoconfirmed users and above. I see we have T266394 for that

Forked the Commons part in T266407 and left this one for the Wikidata part.

In T57755#6576583, @Jc3s5h wrote:

....

There is no indication about what time zone will be assumed for the creation or first publication date. You can't expect users to provide correct dates and times if you don't tell them at the time of upload that whatever they enter will be assumed to have the time zone that was in force at the time and place where the photo was taken.

In T57755#4661653, @Addshore wrote:

So, this ticket was created specifically for allowing entering time values with second precision on wikidata.org and it should probably (i guess) remain just for that.
Having second precision time values on commons does not mean wikidata has to have them.

As said in T57755#4639411 the underlying value object being used already supports second precisions.
Just wikidata has them turned off.
If commons uses the same validators as we use on wikidata.org then second precisions will not work.
If other validators are used then commons can have second precision values.

In T245349#6572376, @Tgr wrote:

Presumably the bot doesn't do an actual null edit but calls the purge API? Those things are supposed to do the same but the actual code path is fairly different.

Thanks for your explanation Gilles. You can see the bug in action at https://commons.wikimedia.org/wiki/Special:NewFiles

@Gilles how did you come up with the number 50? The standard number of thumbnails in a Commons category is 200 and it's very common for galleries to have more thumbnails than that. I've hit this limit many times over the last couple of months. This is a source of much annoyance.

Not sure if this is related: https://commons.wikimedia.org/wiki/Category:Creative_Commons_Attribution-Share_Alike_4.0_International_missing_SDC_copyright_license is empty now, but https://commons.wikimedia.org/w/index.php?sort=last_edit_desc&search=incategory%3ACreative_Commons_Attribution-Share_Alike_4.0_International_missing_SDC_copyright_license&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns6=1&ns9=1&ns12=1&ns14=1&ns100=1&ns106=1 returns over 400.000 hits.

This was just a task for one of the in person events (feels like ages ago!). Online discussion continued on https://commons.wikimedia.org/wiki/Commons:Structured_data/Modeling

In T225897#6557227, @MGA73 wrote:

It seems that structured data breaks old revisions. Not just for random files but for ALL files. So soon we have 60 million files on Commons where the old revisions are broken.

In T245349#6550112, @matthiasmullie wrote:

This (finally - thanks @Tgr!) got merged.
@Jarekt or @Multichill, any chance either of you could verify that this got fixed? (happy to wait if it's too much hassle to try to reproduce!)

Stepping back a bit. Wbeditentity should work more like the normal edit (action=edit) with things like how to handle edit conflicts and also minor edits.

In T122711#6534813, @thiemowmde wrote:

With all the respect, but the task specifically asks to make https://www.wikidata.org/wiki/Q269728.json work. It might be that this is not something the Wikidata team want's to implement and maintain. If this is the case, the task needs to be declined, not closed as "resolved".

I just noticed this mid air collision: https://commons.wikimedia.org/w/index.php?title=File%3ANSG_Salmorth_PM19-09.jpg&type=revision&diff=485124008&oldid=485009736 .
If I understand Adman correctly, adding the option to use baserevid won't solve this. Using wbsetclaim isn't really an option because edits like this would take 8 edits instead of one. Maybe introduce baserevid and some kind of option for the bot to indicate that it wants strict checking instead of a warning?

Raising this to high because more than half the files on Commons have structured data now so the chance of running into this is much higher.
The easiest solution is probably to just show the wikitext in view mode only like when you try to edit a protected page, but you don't have the right to do so. That way we can at least access the wikitext and copy it.

http://www.wikidata.org/entity/Q269728 is the entity uri (not https://www.wikidata.org/wiki/Q269728), http://www.wikidata.org/entity/Q269728.json redirects to https://www.wikidata.org/wiki/Special:EntityData/Q269728.json which returns the json. Marking this one as resolved.

@CDanis based on the webserver logs we should know what domains give the most hits. Can you share a list of these domains which are not on the whitelist already? That would give a good indication of what might be missing before breaking it.

In T192733#6518924, @Xqt wrote:

In T192733#6518857, @Multichill wrote:

In T192733#6516471, @Xqt wrote:

Don't see that this is related to Pywikibot framework.

pywikipedia is the old name for Pywikbot so this is related to the framework, we even have a special swimlane for this on the workboard

Sure but it is related to toolforge. I don’t see anything what can be done on framework side:
https://codesearch.wmcloud.org/pywikibot/?q=trunk%7Crewrite%7Ccompat%20%7Cpywikipedia&i=nope&files=&repos=

In T192733#6516471, @Xqt wrote:

Don't see that this is related to Pywikibot framework.

In T131553#6503636, @Halfak wrote:

Essentially, the idea (if I understand it) is to dedupe commons.

That would imply deleting things. Goal is just to identify images of paintings so that they can be linked to the correct Wikidata item.

I had about 20 bots running and all crashed with errors like:

Ok, I see, we only put misbehaving bots in this group.

It's only on Commons and to reproduce you need a file with no structured data yet. First try doing an edit with "baserevid", you'll get a nasty API error (that's why I'm unable to use that).
To reproduce just do two edits close to each other from different jobs.

IIIF is a whole framework (the F in IIIF). Which API or API's are you planning to work on and what parts? Quite a few of them, see https://iiif.io/technical-details/#information-gathering . In the past we implemented small parts, like https://zoomviewer.toolforge.org/index.php?f=M104_ngc4594_sombrero_galaxy_hi-res.jpg&flash=no as part of T89552 (currently slightly broken). And of course https://mirador.toolforge.org/?manifest=https://wd-image-positions.toolforge.org//iiif/Q1231009/P18/manifest.json is nice too.

Caused by this extension: https://github.com/Smile4ever/link-investigator It will investigate all the links so it will also follow the rollback lists on your watchlist causing a mass revert on all recently edited pages on your watchlist.

In T257938#6451946, @AnneT wrote:

@matthiasmullie any thoughts on getting around the search string character limit here? Now that I've rebased this patch onto the one that swaps out the search modules, I'm hitting the default limit of 300 characters for the cc-by and cc-by-sa filter options, and the "other" option is over the hard limit of 2048 (by about 150 characters, but this could always grow as more license items are added).

Bumping this to high because this makes longer pywikibot bot runs crash.

I think this happens:

• Bot fires up and gets some tokens
• Bot does a lot of edits all with the same token
• For some reason after a long time the token is not valid anymore
• The bot continues to try to edit with the old invalid token instead of getting a new one

How long is a token supposed to be valid? Forever? The Pywikibot code seems to assume that it can use the same token for all edits in a one run. One run can take days or weeks. As a work around I now force the site to get a new token when I run into a problem. That seems to be stable.

I ran into this missing tables. Can the view be added please?

I think I found what is going wrong. site.get_tokens(['csrf']) gets you a new token, but doesn't update the internal token state:

In T261050#6414369, @Tgr wrote:

If the error is easy enough to reproduce, you could try setting the X-Wikimedia-Debug header to get some pretty detailed logs.

I noticed the announcement on the maps-l list and I also noticed https://twitter.com/krmaher/status/1299203640188690434 where @Katherine-WMF replied. Someone recently mention to me "the only way to prevent WMF people from doing making stupid mistakes these days is tagging Katherine on Twitter".

This basically broke the search for me, see https://commons.wikimedia.org/wiki/Commons:Village_pump#cirrusUserTesting=mediasearch_commons_breaks_all_my_queries,_how_to_turn_it_off? .
What is wrong with you people? Why do you unleash poorly tested junk on us without announcing it and without offering the option to opt-out?

In T261050#6410732, @Tgr wrote:

What login method are you using?

In T258835#6407372, @CBogen wrote:

In T258835#6407335, @Multichill wrote:

@CBogen directly using the Commons logo? For Wikidata query service we use https://commons.wikimedia.org/wiki/File:Wikidata_Query_Service_Favicon.svg so you can easily see what tab it is, I would like to have the same for Commons.

That would be ideal, but we don't have the capacity to design a new logo/favicon right now unfortunately.

Technically this is already possible, the question is if we actually want to use it. Makes the data model a lot more complicated to work with.

@CBogen directly using the Commons logo? For Wikidata query service we use https://commons.wikimedia.org/wiki/File:Wikidata_Query_Service_Favicon.svg so you can easily see what tab it is, I would like to have the same for Commons.

@Xqt what is the correct way to force the bot to get a new token? Just site.get_tokens() or something else? I noticed some of my bots now stuck with 100s of errors in a row.

In T258354#6387030, @Ladsgroup wrote:

In T258354#6365838, @Multichill wrote:

Just add a sane (relatively high) ratelimit to the current bots group or keep it as is. The current proposed solution is to try to work around the problem.

Well, that was exactly what we did in T184948: limit page creation and edit rate on Wikidata and you disagreed with it (T184948#4145480)

In T259801#6386751, @Lydia_Pintscher wrote:

Yes I think you're right. But doesn't the same issue apply there?

In T259801#6378907, @Lydia_Pintscher wrote:

This would only affect the UI? How do we imagine this working in exports and APIs?

This bug was filed quite some time ago, but seems it hasn't been noticed yet so tagging some people on it. It just got mentioned again on the village pump on Commons ( https://commons.wikimedia.org/wiki/Commons:Village_pump#Strange_WD_link_below_the_edit_window_of_a_file )

I checked the specs at http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf and looks like exif is at the start of the file. I wonder if you can speed up your bot by just downloading the first part of the file instead of the whole file.

@dschwen usually your bot picks up missing coordinates ( https://commons.wikimedia.org/w/index.php?title=Special:Contributions/DschwenBot&dir=prev&offset=20200806083129&target=DschwenBot ). Any idea why it didn't do it for these example files? I assume it is because you look at the exif info from the database and not from the raw file?

I think the approach is wrong. Let's start with the first assumption. Nuke ( https://www.wikidata.org/wiki/Special:Nuke ) runs under your own account. Used it plenty of times. As an admin I have noratelimit so having noratelimit on a completely different group is not relevant. MassMessage is just an extension that runs in the background. It is a bot. It shouldn't go too fast and it shouldn't crash on ratelimits. Relevant historic bugs: T192690 and T184948 .

Would be two edits. One for the wikitext and one for adding the structured data.

Isn't this something new being rolled out? Would expect, beta, test, group1/2/3. So on https://test.wikipedia.org/w/api.php?action=help&modules=edit I currently see:

In T258895#6340004, @Lucas_Werkmeister_WMDE wrote:

I would really rather not do that. This is a beta service – I think putting it on the official URL shortener whitelist, or even considering to rewrite shortened URLs(!), gives it an undeserved appearance of being more stable than it’s really meant to be.

In T258895#6336041, @Ladsgroup wrote:

Yes, if it goes to *.wikmedia.org or *.commons.org, it'll be shortable in short url but *.wmflabs.org is not among the allowed domain list for security reasons (hiding XSS through open redirects, etc.)

In T257938#6313111, @EBernhardson wrote:

I wonder, would the liceneses extracted from CommonsMetadata be consistent enough to map directly to wikidata properties? It would be a lie, but it seems it would be a cleaner way into the future if we could translate the CommonsMetadata license information into an equivalent wikidata statement, and then add that to the statements indexed.

For adding a new field, ContentHandler::getDataForSearchIndex() calls HookRunner::onSearchDataForIndex(). For whatever reason its SearchEngine that calls HookRunner::onSearchIndexFields() after all content models have had their fields from ContentHandler::getFieldsForSearchIndex() merged.

I don't see any mention of copyright status. A lot of files on Commons don't have a license because these files are in the public domain, see for example https://commons.wikimedia.org/wiki/File:Georges_Ricard-Cordingley_(1873-1939)_-_Deep_Sea_Fishing_(morning)_-_RCIN_406333_-_Royal_Collection.jpg or https://commons.wikimedia.org/wiki/Special:ListFiles/BotMultichillT Not sure how to fit that exactly in the wording. Flickr has something similar, see https://www.flickr.com/search/?text=house going from the most restrictive to the most liberal. I like that approach.

Removed reference to T226453 . This is about Concept URI's on Commons (httpS) and not about concept URI's on Wikidata (http).

In T258590#6331165, @CBogen wrote:

@Multichill it's very possible this discussion was had in the past but that those of us working on it now weren't around at that point - do you happen to have a link to the discussion that we can review? If the decision was already made, then going forward with the change will be a more straightforward next step to take :)

Wait, what, didn't we have this discussion quite some time ago for Commons and decided it would be https from the start? How did the http slip in again? If I look at a not so random item https://commons.wikimedia.org/wiki/Special:EntityData/M90544172.rdf it says :
<schema:about rdf:resource="https://commons.wikimedia.org/entity/M90544172"/>

Code to modify: https://commons.wikimedia.org/wiki/MediaWiki:Gadget-PermissionOTRS.js . Code in https://www.wikidata.org/wiki/MediaWiki:AddQuickClaim.js can probably used as a starting point.

Nevermind, this user is blocked on two Wikipedia's, see https://meta.wikimedia.org/wiki/Special:CentralAuth?target=Lazy-restless

@Lazy-restless care to explain why you re-opened this task? Looks like @Mholloway correctly closed it because all work is done.

Notes for future reference:

Could T250767 be related @Tgr ?

@Lokal_Profil @JeanFred any news on this? Haven't done this in a while and the documentation at https://commons.wikimedia.org/wiki/Commons:Monuments_database/Harvesting looks outdated. Implication is that you two are currently the only people who are able to add sources.

I'm just going to close this one as invalid because of lack of information and nothing happening for several years. Feel free to add more information and open it again.