To link the original file to these objects, three new SDC properties are proposed:

• georeferencing control point data
• georeferencing pixel mask data
• georeferencing mask geoshape

Quick update for those following from home.

So for example this afternoon @Miriam is going to be leading a workshop session introducing image classifiers and clustering algorithms, with a view that Commons users can start to explore writing their own large-scale machine-learning image analysis tools. It would be good if there was syntax in place so they could write the results of those explorations to Commons SDC, for that they were there and accessible for the community to then refine or extract or take further for each image. Similarly two days ago @Multichill led a small group session, to try to think through what was the generic pipeline and workflow for machine-learning contributions, and what kind of open framework was needed to support bulk contributions of that kind from allcomers and any set or subset of images. @Fuzheado too has been talking about some of the investigations he has been doing with machine vision and the Metropolitan Museum collection. All those voices, and more, I think would have useful input for this conversation.

It would be good for unassessed suggestions for statements (and not necessarily just "depicts" statements) to be accessible via the SPARQL query service, in the same way as regular statements, but possibly set with a lower trust level.

Resource page (under development): https://commons.wikimedia.org/wiki/User:Bertspaan/maps

By the way, I have a Wikidata property proposal suggested for external georeferencer URL, since a basic thing we will want to record is whether an external service has georeferencing for an image, and if so what the relevant URL is.

@bert I'll be there; I'm coming in on the Tuesday afternoon flight from Edinburgh, and then I'll be at the Comfort Hotel Xpress Stockholm Central.

@Bugreporter : T225778 "Define canonical URI for EntitySchemas" was opened a few weeks ago as a specific ticket for what the canonical URI should be, so I've added that in as a subtask for this ticket.

A couple of issues:

I do agree with @Legoktm that it would be nice to have a service where one could record and browse one's own and other people's recent queries -- and perhaps tag them into particular categories of interest.

A core point of this service was to be to able to share long queries.

It would be nice -- and not just for depicts statements -- for there to be an additional rank, beyond "preferred", "normal", and "deprecated", so that statements could be given the rank "suggested by machine, but further confirmation desired".

@bert Interesting proposal. But for me it raises some issues. Firstly, about the business case for it. Secondly, regarding implementation.

Agree with @alaa_wmde that no requirement's yet been made out for having structured data on EntitySchemas within Wikidata. But it would be useful to have the EntitySchemas themselves (not surrogate items for them) represented in WDQS, so that they can be queried. That is the subject of ticket T225701 "Add EntitySchemas to the Query Service". A federated service, rather than putting them in the main WDQS triplestore, would probably satisfy this.

^^ I saw exactly what Wurgl reports, browsing long pages on en-wiki (eg the Fram case) in London. Now seems okay.

According to this post on Wikidata-l, there is a standard RDF serialisation (ShExR) for Shape Expressions, with a test suite on the ShEx spec github.

The community has now given the thumbs-up to Wikidata:Property proposal/Shape Expression for class, to link a class item to the Shape Expression that members of it should conform to.

Hi @Smalyshev. You've closed this as resolved, but a query like https://w.wiki/4bn is still returning corrupt data.
How long should it take for the old corrupt values to disappear?

Looks like the issue may be in the code of :c:Module:Artwork, which @Jarekt has been working on. The sandbox version of the module, called by the sandbox version of the template, does not show the problem.

@Tagishsimon the WDQS tag is appropriate though, as it includes the pipeline for getting statements into the WDQS triplestore, and also any corruption issues happening there. Thanks for creating the ticket! #

The other workaround, of course, is just to copy & paste the URL into tinyurl, which seems able to take at least 4500 characters -- but that is something of a step back to square 1.

@Lydia_Pintscher : No, that's T220703 : Maximum length of a URL that can currently be shortened in 2,000 chars; yours was 2,561
Also T221126 (Better error reporting).

I am finding this too.

@GoranSMilovanovic As per @agray : No -- my numbers were counting distinct items with the external IDs, rather than distinct statements, so should exactly match what you're aiming to compute.

@GoranSMilovanovic Overlap table for P 1367 (Art UK Artist ID) now showing about two-thirds of the overlap hits that it should be. (VIAF 6825, ULAN 6153, RKD 5582, ISNI 4240, Benezit 4643) vs true VIAF 11633, ULAN 10552, RKD 9769, ISNI 8237, Benezit 7529.

@Envlh That's very nice. So for example, here are your comparison tables for
P 1367 (Art UK Artist ID): https://tools.dicare.org/properties/?property=1367&type[]=ExternalId
and for P 650 (RKDArtists ID): https://tools.dicare.org/properties/?property=650&type[]=ExternalId

When you've got the data sorted, a table showing the closest identifiers by Jaccard similarity, rather than total overlap, might be quite interesting.

The numbers in the overlap data table for P 1367 (Art UK Artist ID) are way off as well -- only one tenth of the VIAF and ULAN overlaps correctly reported, only one twentieth of the RKD Artist ID overlaps.

Why are we baking in the assumption that only one Wikibase instance can be associated with a particular entity type?

As a postscript to my comment two posts above, note that in such a scenario a Commons category page might well be associated with both an item on Wikidata (via a sitelink equivalence) and a local item on the Commons wikibase.

See also: T180113 "Support the creation and use of volunteer tools that help to convert information in Commons categories to structured data"

In the context of this thread, it's worth recalling the ongoing wish from Commons users for Commons categories to be able to have their own local items on the Commons wikibase.

Hi @JeanFred . The point is not so much to link multiple versions of the same viewer, but rather to link to some of the different viewers available, for example here is the Alba Madonna in the "Universal Viewer" used as a default by the Wellcome Collection and the British Library and others.

Note that there has been some discussion on the IIIF mailing list as to whether a https://www.wikidata.org/wiki/Property:P1630 URL formatter should be added for the property:

To support what Smalyshev said: occasional termporary update lag may not be such a high-priority issue; but prolonged or repeated update lag rapidly would be.

Very nice!

That's nice. I didn't know that service included Commons.

(Copied from T173346)
I'd like to note strong interest on two different occasions from two different people in the last month in a top-tier GLAM that I'm working with, about the possibility of being able to use Commons as an IIIF hosting service for high-resolution images, eg for serving tiles of old maps for geo-referencing to platforms that need to be able to access images from an IIIF service.

Not sure if this is the right ticket for this information, but I'd like to note strong interest on two different occasions from two different people in the last month in a top-tier GLAM that I'm working with, about the possibility of being able to use Commons as an IIIF hosting service for high-resolution images, eg for serving tiles of old maps for geo-referencing to platforms that need to be able to access images from an IIIF service.

Presumably (as the original ticket suggests), one option would be to set up a Commons Data Query Service (CDQS) as a distinct endpoint on a new server that had both Wikidata and CommonsData installed in a Blazegraph instance locally. This would allow the existing WDQS service to continue without alteration, while allowing the CDQS service set-up to be optimised, as we learn more about the sort of queries that people want to run.

The 400 character limit can also be a problem for the titles of old books, maps, etc -- the original titles of these, as given eg in library catalogues, can get pretty long.

See this query: tinyurl.com/y76l2bft for an example of how to draw lines on a WDQS results map, based on values from variables in a WDQS query (in this case, drawing the bounding boxes of some maps)

Thanks @SandraF_WMF . I've started putting together some links at :c:User:Jheald/IIIF_180815 that I or Andy Mabbett could talk through, to give an idea of what sort of IIIF interaction is possible at the moment.

A heads-up that "Wikipedia and IIIF" is the proposed subject for the IIIF community call this week -- see https://groups.google.com/forum/?hl=en#!topic/iiif-discuss/wy2uRl_ukJ0

Interesting slide-show. But the fundamental problem -- as some of the attached tickets start to appreciate -- is that the key information that determines whether an image fits the specification being looked for is not going to be stored in statements just on the CommonsData item for the image in question, nor just on the Wikidata item for the thing it depicts, but is going to depend on statements distributed throughout the database.

Well of course you're going to copy all the CommonsData statements to Blazegraph.

I have now found T198261 and T199119, which investigate how some of the drawbacks above with the simple string-matching approach might be addressed.

I'd say don't give up too easily. This is probably as good an approach as any. If the issues are structural, bots will fall prey to them in just the same way, just more slowly and more haphazardly.

Don't you think you should maybe talk to the community about this first ?

Another example, where such image searches may depend quite sensitively on query construction or query optimisation: Category:Grade I listed buildings in Bedfordshire.

The attached subtickets are an interesting read. They all seem to be based on taking the Q-number value of "depicts", storing it as a string in the text-search index, and then doing an indexed string-match for it. Of course first baby-steps are important, and this facility will be crucial to be able to confirm correct entry, storage, and direct retrievability of "depicts" values.

One sidelight on this. QS2 is great (invaluable!) for adding new statements, but not so good for modifying existing statements, especially if they are currently heavily qualified or referenced -- at the moment, the entire statement complete with all qualifiers and references has to be re-created, even if one only wants to change one qualifier value, or migrate the property being used for the qualifier.

IIIF image extracts seem to be broken again: see eg the 'Virgin amongst the virgins' test page at Crotos

Hi Magnus,

That looks so great now. Thank you! (And @Shonagon ).

@dschwen Any idea why the IIIF-based detail viewer is working for the images on this page of Shonagon's:

• Vierge entre les vierges,

It would be good to make this work.

+1 on the need for this.

To make this really useful, it would be valuable for a WDQS query to be able to display such shapes in the map view of query results.

So you're saying, in effect, I should think of the strings being stored as a great big hash table rather than a B-tree, so there's nothing there that can help even STRSTARTS. And of course I know very little about the internals of BlazeGraph, whereas you've actually written modules for it. But I did think BlazeGraph uses B+ trees, which do specifically facilitate rapid in-order traversal and retrieval. But perhaps only to retrieve hashes, not strings?

Thanks for your speedy diagnosis. I've gone through and reverted and then un-reverted the ten edits by hand, so they are now fine.

Ouch. That sounds quite nasty. The only thing I can think of from the user side that was perhaps slightly different about this set of edits was they were made with QuickStatements while I already had a different QuickStatements run open and going in another window (a big batch of 10,000 corrections in capitalization of item labels, which took about two and a half hours to run, all of which went through fine).

Hi Smalyshev, thanks for the comment; but if I can come back on your two objections:

Perfect. I would never ever have found that.

Another case where it could be useful to be able to cast calculated co-ordinates back to a geo:wktLiteral would be to allow AVG() co-ordinates to be calculated and plotted, in cases where SAMPLE() is not sufficient.

Trying to get round this, by just plotting the values in a scatterplot, without a map, also failed -- though that's likely to be an issue with the scatterplot routine. (Ticket T160325 opened).

Example case:

As one possible hack that could be a way forward on this, I see that the Blazegraph GeoSpatial extension (description on BlazeGraph wiki) allows custom compound data-types to be defined -- which do not necessarily need to include a geospatial element.

As an aside, what I really wanted to do was to plot the values on a map.

Indeed, I still get the same behaviour even if I dispense with the third column entirely, and SELECT just the first two (query) -- so the issue may not involve the counts at all.

Example query (view in query editor):

Thanks for this Jonas, but given there is already the nice viewer on the page (which I didn't know about when I wrote the bug, because until then I'd only right-clicked on it), perhaps on consideration a better link is just the simple Commons file page, ie 'https://commons.wikimedia.org/wiki/File:{FILENAME}'

+1 on this.

Re: my previous comment, nodes that can store both a string and a URL would appear to be necessary to enable T121274 "Provide an RDF mapping for external identifiers" -- even though there are no such value nodes at the moment.

So something like p:P569/psv:P569 returns a URI, something like wdv:8f6e57348b9035361151ee05475253ef

So is there no way to create any kind of compound type that the GUI can interpret appropriately?
-eg date + precision, here
-or url + linktext, on other tickets

So if instead one wrote a query to return a wikibase:Time,
-eg by using p:P569/psv:P569 instead of wdt:P569
could it be possible for the GUI to pick that up and return an appropriately precisioned time?

It would also be useful for the GUI to show Wikipedia sitelinks as links rather than full URLs.

and T148245 "Explore making WDQS access mediawiki API"

See also T157676 "Provide access to category information from WDQS SPARQL"

You're right. I see that now.

• https://www.w3.org/TR/sparql11-query/#scopeFilters