In T282222#7069792, @dcausse wrote:

My bad, the depool command I ran this morning did not work due to an error running the command (I ran sudo depool wdqs1012 instead of sudo depool so it depooled the service wdqs1012 which is nonexistent).

For the record I ran the proper command now:

dcausse@wdqs1012:~$ sudo depool
Depooling all services on wdqs1012.eqiad.wmnet
eqiad/wdqs/wdqs/wdqs1012.eqiad.wmnet: pooled changed yes => no
eqiad/wdqs/wdqs-heavy-queries/wdqs1012.eqiad.wmnet: pooled changed yes => no
eqiad/wdqs/wdqs-ssl/wdqs1012.eqiad.wmnet: pooled changed yes => no

This server should stop receiving traffic now.

As a work around on pages like https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings/Old_European_art_missing_genre/Sweden where most thumbnails don't work:
wget -E -H -k -K -p https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings/Old_European_art_missing_genre/Sweden

File names are bad URI's. Files get renamed all the time (see https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=&limit=500&type=move ) causing all sorts of breakage. The pageid stays the same so the mediaid also stays the same. That's a much more stable identifier.

In T258776#6921752, @CBogen wrote:

I'm going to close this as a duplicate of T277665 as that is what we see as the practical solution to this problem. Please re-open if you don't think that will solve your use case.

In T258590#6363261, @CBogen wrote:

Note that the SD team work to change the Concept URIs in Commons is estimated to be a small.

In T166094#6878436, @Ainali wrote:

In T166094#6878327, @Multichill wrote:

No we don't, captions are just labels, see https://commons.wikimedia.org/wiki/Special:EntityData/M96527.json . Description is currently unused in SDOC.

Well, perhaps the variable is called label in the code, but in the Wikipedia app, people are being asked to fill these fields as image captions. So it's a bit of a conflicting usage of that specific field right now (descriptions on Commons, captions in the Wikipedia app), but neither of them are for the purpose of alt text. Which means it would be a huge work cleaning up before it would be useable even if we started to designate that field for it. Better start fresh in my opinion (and if the label/description/caption field is redundant in your opinion, then suggest remove it in a separate ticket).

In T166094#6877045, @Ainali wrote:

Well, then we need a new field for the caption. They are not meant to solve the same problem and should almost always be different from each other. So for every image it would be very useful with two separate text fields.

Why are we not using the description field for this? Seems more sensible to me than creating new properties

The URI for the image is https://commons.wikimedia.org/entity/M6919529 (yes, https, not http, that got messed up, see T258590). You can see an RDF representation of that at https://commons.wikimedia.org/entity/M6919529.rdf . We should someone include this URI on Wikidata too. The royal way would be to update the image data type which accepts the Mediainfo ID and does all the logic like the current image data type.

In T275787#6871707, @CBogen wrote:

In T275787#6866509, @Multichill wrote:

@CBogen why did you remove the search projects? This looks very search to me.

The Discovery-Search project is mainly for backend search work; this is a front-end project in MediaSearch on Commons so it would be part of the Structured Data team's purview.

We now have more than 1 million files with location of creation on Commons, see https://commons.wikimedia.org/w/index.php?search=haswbstatement%3AP1071&title=Special%3ASearch

If it aint' broken, don't fix it? Let's just see what happens and if anything explodes, than focus on fixing that.

Wrote https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings/Automated_image_uploads and shared the link by email some time ago.

In T271105#6814507, @Lucas_Werkmeister_WMDE wrote:

In T271105#6812288, @Multichill wrote:

This change is subject to the https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy . Please complete the steps listed there.

Can you clarify which part of the policy you think hasn’t been followed so far?

This change is subject to the https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy . Please complete the steps listed there.

I obviously like getting more structured data and better usage, but I see some issues with this task: The scope seems to be very broad. Is this intentional? What kind of time investment is expected? Might have a higher chance of making a difference when it's more tightly scoped. We have several steps in the process and possible tasks related to it:

• Convert existing data from wikitext to structured data. This is already happening on a small subset, but on a large scale. Not controversial as long as you don't remove any data. Still a ton of work to do here, but quite a few things are not clear yet on how to model things. Building some kind of workflow to extract knowledge from the category tree and show it to users for approval would be extremely useful.
• Show the structured data in a pretty way. You already mentioned some examples, https://commons.wikimedia.org/wiki/Module:Artwork and https://commons.wikimedia.org/wiki/Module:Information . The current approach here is to do incremental improvements without changing the look and view from what it used to be. You can also take a radical different approach like skins: You make another implementation for the same data with a completely different look and feel. With some logic logged-in users can enable this new template skin. That way you can experiment quite freely without disturbing the Monobook people. This task would be fun for someone who is on the edge of development and design.

Reasoning why the convention is suddenly enforced are missing. Instructions on how to fix are also missing.

In T267175#6754527, @dcausse wrote:

Checked a couple of these inconsistencies and they appear to all be out of order in the kafka topics. I suggest to disable async imports as I believe it might be possible cause of these inconsistencies.

@RKemper so what's the status of this? I see a lot of cases where the last edit didn't get processed so the data in SPARQL is not consistent. See https://w.wiki/ugf for some examples.

In T180113#6747475, @Elitre wrote:

@CBogen Hi, can you help me figure out if this is needed, otherwise I'd like to resolve. Thanks.

Just had it again:
pywikibot.data.api.APIMWException: internal_api_error_JobQueueError: [X-VEtQpAIDkAAHaGUqkAAADW] Caught exception of type JobQueueError
[servedby: mw1345;
errorclass: JobQueueError]

Ha, right after I posted that my bot crashed twice. Now with internal API errors:

One of my robots ran non-stop for the last week so looks like it's not happening at the moment. You got to love intermittent problems.....

https://commons.wikimedia.org/wiki/Commons:Bureaucrats%27_noticeboard#Remove_users_from_gwtoolset_group_who_haven%27t_used_it_in_the_last_two_years

Thanks Sam, can you do a per user per year breakdown? I expect a small number of users with a lot of uploads.

I've observed quite a lot of inconsistencies over the past two weeks. I haven't looked very extensively at it, but I'm getting the impression that blocks of edits are missed and the timestamps are around a spike at https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&viewPanel=8 . Order of magnitude is several 100s of edits.

@Lydia_Pintscher @Aklapper this is a never ending story

Tested on production and works as expected. See for example https://commons.wikimedia.org/w/index.php?title=File:John_Quinton_Pringle_(1864-1925)_-_Poultry_Yard,_Gartcosh_-_GMA_37_-_National_Galleries_of_Scotland.jpg&action=edit&oldid=518956340

In T225897#6665615, @Etonkovidova wrote:

Checked in production (commons wmf.20) and in betalabs commons (tested more in depth there).

In T131553#6647417, @mat_kelcey wrote:

e,g, how you generated "https://tools.wmflabs.org/multichill/queries2/commons/paintings_without_wikidata_in_painting_category.txt"

In T268645#6645735, @Xqt wrote:

Looks like it is the same as T192912

Due to

if code == 'failed-save' and \
   self.action == 'wbeditentity' and \
   self._is_wikibase_error_retryable(result['error']):
    self.wait()
    continue

it seems that action is not 'wbeditentity'; any idea what it was?

In T257938#6645626, @Cparle wrote:

@AnneT the config patch has been merged so you're good to go on the UI patch. Options are

• haslicense:cc-by
• haslicense:cc-by-sa
• haslicense:unrestricted
• haslicense:other

Hi folks, happy to see activity on this task. Current code at https://github.com/multichill/toollabs/blob/master/bot/wikidata/find_painting_images.py#L704

@Cparle @matthiasmullie I noticed the change was merged. Do you have a pointer where the actual mappings are stored? See https://github.com/multichill/toollabs/blob/master/bot/commons/own_work_sdoc.py#L66 for a list of mappings that I would expect (minus the variants).

Thanks for adding screen. Can you add /usr/bin/mysql to the whitelist too? Just the client part. Mariadb server will auto disconnect any long open sessions and the client will just reconnect when a session is needed again.

I just noticed this also breaks https://commons.wikimedia.org/wiki/Special:MediaSearch if you sort it by "recency".

I looked around in old bugs and found T129046 . I think it went like this:

• Several Wikibase page_props were added for some unknown reason (you would have to look in the code where these are actually used now)
• I wanted to track items without statements (T129037) and easiest was to use the page_props (T129046)
• Some other page_props were added to the RDF later

Very confusing indeed. I updated https://www.wikidata.org/wiki/MediaWiki:Protect-text

Still means the wording of the protect page is incorrect. Currently it says "all users" and "allow only administrators". The "all users" is not correct because changing the protection to that won't make it possible for all users to edit, only for autoconfirmed users and above. I see we have T266394 for that

Forked the Commons part in T266407 and left this one for the Wikidata part.

In T57755#6576583, @Jc3s5h wrote:

....

There is no indication about what time zone will be assumed for the creation or first publication date. You can't expect users to provide correct dates and times if you don't tell them at the time of upload that whatever they enter will be assumed to have the time zone that was in force at the time and place where the photo was taken.

In T57755#4661653, @Addshore wrote:

So, this ticket was created specifically for allowing entering time values with second precision on wikidata.org and it should probably (i guess) remain just for that.
Having second precision time values on commons does not mean wikidata has to have them.

As said in T57755#4639411 the underlying value object being used already supports second precisions.
Just wikidata has them turned off.
If commons uses the same validators as we use on wikidata.org then second precisions will not work.
If other validators are used then commons can have second precision values.

In T245349#6572376, @Tgr wrote:

Presumably the bot doesn't do an actual null edit but calls the purge API? Those things are supposed to do the same but the actual code path is fairly different.

Thanks for your explanation Gilles. You can see the bug in action at https://commons.wikimedia.org/wiki/Special:NewFiles

@Gilles how did you come up with the number 50? The standard number of thumbnails in a Commons category is 200 and it's very common for galleries to have more thumbnails than that. I've hit this limit many times over the last couple of months. This is a source of much annoyance.

Not sure if this is related: https://commons.wikimedia.org/wiki/Category:Creative_Commons_Attribution-Share_Alike_4.0_International_missing_SDC_copyright_license is empty now, but https://commons.wikimedia.org/w/index.php?sort=last_edit_desc&search=incategory%3ACreative_Commons_Attribution-Share_Alike_4.0_International_missing_SDC_copyright_license&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns6=1&ns9=1&ns12=1&ns14=1&ns100=1&ns106=1 returns over 400.000 hits.

This was just a task for one of the in person events (feels like ages ago!). Online discussion continued on https://commons.wikimedia.org/wiki/Commons:Structured_data/Modeling

In T225897#6557227, @MGA73 wrote:

It seems that structured data breaks old revisions. Not just for random files but for ALL files. So soon we have 60 million files on Commons where the old revisions are broken.

In T245349#6550112, @matthiasmullie wrote:

This (finally - thanks @Tgr!) got merged.
@Jarekt or @Multichill, any chance either of you could verify that this got fixed? (happy to wait if it's too much hassle to try to reproduce!)

Stepping back a bit. Wbeditentity should work more like the normal edit (action=edit) with things like how to handle edit conflicts and also minor edits.

In T122711#6534813, @thiemowmde wrote:

With all the respect, but the task specifically asks to make https://www.wikidata.org/wiki/Q269728.json work. It might be that this is not something the Wikidata team want's to implement and maintain. If this is the case, the task needs to be declined, not closed as "resolved".

I just noticed this mid air collision: https://commons.wikimedia.org/w/index.php?title=File%3ANSG_Salmorth_PM19-09.jpg&type=revision&diff=485124008&oldid=485009736 .
If I understand Adman correctly, adding the option to use baserevid won't solve this. Using wbsetclaim isn't really an option because edits like this would take 8 edits instead of one. Maybe introduce baserevid and some kind of option for the bot to indicate that it wants strict checking instead of a warning?

Raising this to high because more than half the files on Commons have structured data now so the chance of running into this is much higher.
The easiest solution is probably to just show the wikitext in view mode only like when you try to edit a protected page, but you don't have the right to do so. That way we can at least access the wikitext and copy it.

http://www.wikidata.org/entity/Q269728 is the entity uri (not https://www.wikidata.org/wiki/Q269728), http://www.wikidata.org/entity/Q269728.json redirects to https://www.wikidata.org/wiki/Special:EntityData/Q269728.json which returns the json. Marking this one as resolved.

@CDanis based on the webserver logs we should know what domains give the most hits. Can you share a list of these domains which are not on the whitelist already? That would give a good indication of what might be missing before breaking it.

In T192733#6518924, @Xqt wrote:

In T192733#6518857, @Multichill wrote:

In T192733#6516471, @Xqt wrote:

Don't see that this is related to Pywikibot framework.

pywikipedia is the old name for Pywikbot so this is related to the framework, we even have a special swimlane for this on the workboard

Sure but it is related to toolforge. I don’t see anything what can be done on framework side:
https://codesearch.wmcloud.org/pywikibot/?q=trunk%7Crewrite%7Ccompat%20%7Cpywikipedia&i=nope&files=&repos=

In T192733#6516471, @Xqt wrote:

Don't see that this is related to Pywikibot framework.

In T131553#6503636, @Halfak wrote:

Essentially, the idea (if I understand it) is to dedupe commons.

That would imply deleting things. Goal is just to identify images of paintings so that they can be linked to the correct Wikidata item.

I had about 20 bots running and all crashed with errors like:

Ok, I see, we only put misbehaving bots in this group.

It's only on Commons and to reproduce you need a file with no structured data yet. First try doing an edit with "baserevid", you'll get a nasty API error (that's why I'm unable to use that).
To reproduce just do two edits close to each other from different jobs.

IIIF is a whole framework (the F in IIIF). Which API or API's are you planning to work on and what parts? Quite a few of them, see https://iiif.io/technical-details/#information-gathering . In the past we implemented small parts, like https://zoomviewer.toolforge.org/index.php?f=M104_ngc4594_sombrero_galaxy_hi-res.jpg&flash=no as part of T89552 (currently slightly broken). And of course https://mirador.toolforge.org/?manifest=https://wd-image-positions.toolforge.org//iiif/Q1231009/P18/manifest.json is nice too.

Caused by this extension: https://github.com/Smile4ever/link-investigator It will investigate all the links so it will also follow the rollback lists on your watchlist causing a mass revert on all recently edited pages on your watchlist.

In T257938#6451946, @AnneT wrote:

@matthiasmullie any thoughts on getting around the search string character limit here? Now that I've rebased this patch onto the one that swaps out the search modules, I'm hitting the default limit of 300 characters for the cc-by and cc-by-sa filter options, and the "other" option is over the hard limit of 2048 (by about 150 characters, but this could always grow as more license items are added).

Bumping this to high because this makes longer pywikibot bot runs crash.

I think this happens:

• Bot fires up and gets some tokens
• Bot does a lot of edits all with the same token
• For some reason after a long time the token is not valid anymore
• The bot continues to try to edit with the old invalid token instead of getting a new one

How long is a token supposed to be valid? Forever? The Pywikibot code seems to assume that it can use the same token for all edits in a one run. One run can take days or weeks. As a work around I now force the site to get a new token when I run into a problem. That seems to be stable.

I ran into this missing tables. Can the view be added please?

I think I found what is going wrong. site.get_tokens(['csrf']) gets you a new token, but doesn't update the internal token state:

In T261050#6414369, @Tgr wrote:

If the error is easy enough to reproduce, you could try setting the X-Wikimedia-Debug header to get some pretty detailed logs.

I noticed the announcement on the maps-l list and I also noticed https://twitter.com/krmaher/status/1299203640188690434 where @Katherine-WMF replied. Someone recently mention to me "the only way to prevent WMF people from doing making stupid mistakes these days is tagging Katherine on Twitter".