@Pintoch That's great. I'' try and get matching these this weekend. Is there any chance of the full dataset, beyond the first 1000? We currently have 1660 items for bodies of water in the UK (plus more that possibly don't have P31s), so it would be nice to be able to try to match them all. But thanks again for this!

If maxlag is to be based on the maximum lag of the pooled servers, will there be active measures to monitor these, and take any really badly lagged server (ie significantly worse lagged than any of the others) out of the pool, and out of the maxlag calculation, to give it a chance to recover ?

I would love it if somebody would take this on. I think it's a really worthwhile resource to link to, and a really significant topic to improve our data for.

Hi @Pintoch. Thanks for this.

One thing that seems odd (to an outsider like me who knows very little about the system) is that some servers seem to be performing so much worse than others.

@Lucas_Werkmeister_WMDE How dependent is the wikibase constraint system on SPARQL ?
Maarten was just suggesting that a functioning SPARQL service is required for any of the constraint checking to operate (and so this ticket would be completely blocked until CDQS is reliably up and running)
Is that correct? Or are any of the simplest of the constraint checks (eg format constraints, one-of constraints, etc) available without SPARQL ?

I very strongly agree with this ticket. It needs to be as easy as possible for query-service users to find queries that illustrate "how do I do this?"

Use-case example to show why this is urgently needed: The property source of file (P7482) is intended to show the broad nature of the origin of a file.

Example: property source of file (P7482) is intended to show the broad nature of the origin of a file.

@Lydia_Pintscher - Having now kicked a few possibilities around on Project Chat, can we go for creating badges with:

There is some sense in what MisterSynergy says, but I also think there is sense in what @deryckchan wrote in the RfC (here), namely that there still ought to be some warning if someone tries to sitelink to a redirect page, and the user should be made to actively confirm that they didn't want to instead link to the redirect target. Otherwise I could see us ending up with a lot of accidental sitelinks to mis-spellings, which the present system mostly keeps us safe from.

Thanks Lydia. I've started a thread at Wikidata:Project_chat#Badges_for_sitelinks_to_redirects to quickly see if there are particular icons people would prefer; and created two items, Q70893996 ("sitelink to redirect") and Q70894304 ("intentional sitelink to redirect") where we can start to assemble translations.

The badges, and corresponding bot, might be a nice quick win for Wikidata's 7th birthday.

Mostly, the existence of a sitelink to a redirect indicates a potential data problem on Wikidata: a sitelink that has been left over when two Wikipedia articles have been merged, but no corresponding merge has been made on Wikidata. A sitelink to a redirect can therefore be a strong indication that an item on Wikidata is a duplicate of another one, and should be merged with it. However, at the moment it is not possible for somebody browsing a Wikidata article to readily spot that a sitelink points to a redirect. A badge to identify this would bring the fact into plain sight.

For discoverability, maintenance, and reuse, it is as important to be able to store metadata for datasets via SDC as it is to be able to store metadata for images.

Are we any further forward with this?

On a separate but related issue: we now have quite a lot of images of old maps on Commons, with coordinate georeferencing allowing the maps to be "warped" to standard coordinate systems. It would be nice to be able to serve the warped versions of the maps as tilesets, allowing users to compare different historical representations of given places as layers.

Yeah, this is a problem. I recently had a new Wikidata item (Q66458942) deleted, because the admin wasn't able to see that it was being used as a value in SDC statements on several files, even though he'd checked https://commons.wikimedia.org/wiki/Special:EntityUsage/Q66458942 (cf discussion on admin's talk page).

The aim was to try to identify the different generic stages of the workflow in machine vision, and to think how each stage could be made as "pluggable" as possible, so how users could be made able to plug in their own approach at any stage, and report their results from that stage back into the overall pipeline.

To link the original file to these objects, three new SDC properties are proposed:

• georeferencing control point data
• georeferencing pixel mask data
• georeferencing mask geoshape

Quick update for those following from home.

So for example this afternoon @Miriam is going to be leading a workshop session introducing image classifiers and clustering algorithms, with a view that Commons users can start to explore writing their own large-scale machine-learning image analysis tools. It would be good if there was syntax in place so they could write the results of those explorations to Commons SDC, for that they were there and accessible for the community to then refine or extract or take further for each image. Similarly two days ago @Multichill led a small group session, to try to think through what was the generic pipeline and workflow for machine-learning contributions, and what kind of open framework was needed to support bulk contributions of that kind from allcomers and any set or subset of images. @Fuzheado too has been talking about some of the investigations he has been doing with machine vision and the Metropolitan Museum collection. All those voices, and more, I think would have useful input for this conversation.

It would be good for unassessed suggestions for statements (and not necessarily just "depicts" statements) to be accessible via the SPARQL query service, in the same way as regular statements, but possibly set with a lower trust level.

Resource page (under development): https://commons.wikimedia.org/wiki/User:Bertspaan/maps

By the way, I have a Wikidata property proposal suggested for external georeferencer URL, since a basic thing we will want to record is whether an external service has georeferencing for an image, and if so what the relevant URL is.

@bert I'll be there; I'm coming in on the Tuesday afternoon flight from Edinburgh, and then I'll be at the Comfort Hotel Xpress Stockholm Central.

@Bugreporter : T225778 "Define canonical URI for EntitySchemas" was opened a few weeks ago as a specific ticket for what the canonical URI should be, so I've added that in as a subtask for this ticket.

A couple of issues:

I do agree with @Legoktm that it would be nice to have a service where one could record and browse one's own and other people's recent queries -- and perhaps tag them into particular categories of interest.

A core point of this service was to be to able to share long queries. (I'm guessing WDQS now accounts for the majority of organically requested shortenings?)

It would be nice -- and not just for depicts statements -- for there to be an additional rank, beyond "preferred", "normal", and "deprecated", so that statements could be given the rank "suggested by machine, but further confirmation desired".

@bert Interesting proposal. But for me it raises some issues. Firstly, about the business case for it. Secondly, regarding implementation.

Agree with @alaa_wmde that no requirement's yet been made out for having structured data on EntitySchemas within Wikidata. But it would be useful to have the EntitySchemas themselves (not surrogate items for them) represented in WDQS, so that they can be queried. That is the subject of ticket T225701 "Add EntitySchemas to the Query Service". A federated service, rather than putting them in the main WDQS triplestore, would probably satisfy this.

^^ I saw exactly what Wurgl reports, browsing long pages on en-wiki (eg the Fram case) in London. Now seems okay.

According to this post on Wikidata-l, there is a standard RDF serialisation (ShExR) for Shape Expressions, with a test suite on the ShEx spec github.

The community has now given the thumbs-up to Wikidata:Property proposal/Shape Expression for class, to link a class item to the Shape Expression that members of it should conform to.

Hi @Smalyshev. You've closed this as resolved, but a query like https://w.wiki/4bn is still returning corrupt data.
How long should it take for the old corrupt values to disappear?

Looks like the issue may be in the code of :c:Module:Artwork, which @Jarekt has been working on. The sandbox version of the module, called by the sandbox version of the template, does not show the problem.

@Tagishsimon the WDQS tag is appropriate though, as it includes the pipeline for getting statements into the WDQS triplestore, and also any corruption issues happening there. Thanks for creating the ticket! #

The other workaround, of course, is just to copy & paste the URL into tinyurl, which seems able to take at least 4500 characters -- but that is something of a step back to square 1.

@Lydia_Pintscher : No, that's T220703 : Maximum length of a URL that can currently be shortened in 2,000 chars; yours was 2,561
Also T221126 (Better error reporting).

I am finding this too.

@GoranSMilovanovic As per @agray : No -- my numbers were counting distinct items with the external IDs, rather than distinct statements, so should exactly match what you're aiming to compute.

@GoranSMilovanovic Overlap table for P 1367 (Art UK Artist ID) now showing about two-thirds of the overlap hits that it should be. (VIAF 6825, ULAN 6153, RKD 5582, ISNI 4240, Benezit 4643) vs true VIAF 11633, ULAN 10552, RKD 9769, ISNI 8237, Benezit 7529.

@Envlh That's very nice. So for example, here are your comparison tables for
P 1367 (Art UK Artist ID): https://tools.dicare.org/properties/?property=1367&type[]=ExternalId
and for P 650 (RKDArtists ID): https://tools.dicare.org/properties/?property=650&type[]=ExternalId

When you've got the data sorted, a table showing the closest identifiers by Jaccard similarity, rather than total overlap, might be quite interesting.

The numbers in the overlap data table for P 1367 (Art UK Artist ID) are way off as well -- only one tenth of the VIAF and ULAN overlaps correctly reported, only one twentieth of the RKD Artist ID overlaps.

Why are we baking in the assumption that only one Wikibase instance can be associated with a particular entity type?

As a postscript to my comment two posts above, note that in such a scenario a Commons category page might well be associated with both an item on Wikidata (via a sitelink equivalence) and a local item on the Commons wikibase.

See also: T180113 "Support the creation and use of volunteer tools that help to convert information in Commons categories to structured data"

In the context of this thread, it's worth recalling the ongoing wish from Commons users for Commons categories to be able to have their own local items on the Commons wikibase.

Hi @JeanFred . The point is not so much to link multiple versions of the same viewer, but rather to link to some of the different viewers available, for example here is the Alba Madonna in the "Universal Viewer" used as a default by the Wellcome Collection and the British Library and others.

Note that there has been some discussion on the IIIF mailing list as to whether a https://www.wikidata.org/wiki/Property:P1630 URL formatter should be added for the property:

To support what Smalyshev said: occasional termporary update lag may not be such a high-priority issue; but prolonged or repeated update lag rapidly would be.

Very nice!

That's nice. I didn't know that service included Commons.

(Copied from T173346)
I'd like to note strong interest on two different occasions from two different people in the last month in a top-tier GLAM that I'm working with, about the possibility of being able to use Commons as an IIIF hosting service for high-resolution images, eg for serving tiles of old maps for geo-referencing to platforms that need to be able to access images from an IIIF service.

Not sure if this is the right ticket for this information, but I'd like to note strong interest on two different occasions from two different people in the last month in a top-tier GLAM that I'm working with, about the possibility of being able to use Commons as an IIIF hosting service for high-resolution images, eg for serving tiles of old maps for geo-referencing to platforms that need to be able to access images from an IIIF service.

Presumably (as the original ticket suggests), one option would be to set up a Commons Data Query Service (CDQS) as a distinct endpoint on a new server that had both Wikidata and CommonsData installed in a Blazegraph instance locally. This would allow the existing WDQS service to continue without alteration, while allowing the CDQS service set-up to be optimised, as we learn more about the sort of queries that people want to run.

The 400 character limit can also be a problem for the titles of old books, maps, etc -- the original titles of these, as given eg in library catalogues, can get pretty long.

See this query: tinyurl.com/y76l2bft for an example of how to draw lines on a WDQS results map, based on values from variables in a WDQS query (in this case, drawing the bounding boxes of some maps)

Thanks @SandraF_WMF . I've started putting together some links at :c:User:Jheald/IIIF_180815 that I or Andy Mabbett could talk through, to give an idea of what sort of IIIF interaction is possible at the moment.

A heads-up that "Wikipedia and IIIF" is the proposed subject for the IIIF community call this week -- see https://groups.google.com/forum/?hl=en#!topic/iiif-discuss/wy2uRl_ukJ0

Interesting slide-show. But the fundamental problem -- as some of the attached tickets start to appreciate -- is that the key information that determines whether an image fits the specification being looked for is not going to be stored in statements just on the CommonsData item for the image in question, nor just on the Wikidata item for the thing it depicts, but is going to depend on statements distributed throughout the database.

Well of course you're going to copy all the CommonsData statements to Blazegraph.

I have now found T198261 and T199119, which investigate how some of the drawbacks above with the simple string-matching approach might be addressed.

I'd say don't give up too easily. This is probably as good an approach as any. If the issues are structural, bots will fall prey to them in just the same way, just more slowly and more haphazardly.

Don't you think you should maybe talk to the community about this first ?

Another example, where such image searches may depend quite sensitively on query construction or query optimisation: Category:Grade I listed buildings in Bedfordshire.

The attached subtickets are an interesting read. They all seem to be based on taking the Q-number value of "depicts", storing it as a string in the text-search index, and then doing an indexed string-match for it. Of course first baby-steps are important, and this facility will be crucial to be able to confirm correct entry, storage, and direct retrievability of "depicts" values.

One sidelight on this. QS2 is great (invaluable!) for adding new statements, but not so good for modifying existing statements, especially if they are currently heavily qualified or referenced -- at the moment, the entire statement complete with all qualifiers and references has to be re-created, even if one only wants to change one qualifier value, or migrate the property being used for the qualifier.

IIIF image extracts seem to be broken again: see eg the 'Virgin amongst the virgins' test page at Crotos

Hi Magnus,

That looks so great now. Thank you! (And @Shonagon ).

@dschwen Any idea why the IIIF-based detail viewer is working for the images on this page of Shonagon's:

• Vierge entre les vierges,

It would be good to make this work.