Many files on Commons cannot be found in WCQS
Closed, ResolvedPublicBUG REPORT
Actions

Description

Many files on Wikimedia Commons which have structured claims cannot be found in Wikimedia Commons Query Service.

One example is https://commons.wikimedia.org/wiki/File:Dux-Markt-1.jpg. It has this concept URI http://commons.wikimedia.org/entity/M47869727 and it has structured data since 30 September 2020. But this query in WCQS has no results:

SELECT *
WHERE
{
  sdc:M47869727 ?predicate ?object.
}

Many files are missing in WCQS. This query finds some images on Wikidata, and then tries to find the mediaInfo entities in WCQS:

SELECT (COUNT (DISTINCT ?image) AS ?images) (COUNT(DISTINCT ?file) AS ?files)
WITH
{
  SELECT ?image ?contentUrl
  WHERE
  {
    SERVICE <https://query.wikidata.org/sparql>
    {
      ?item wdt:P31 wd:Q5153359 .
      ?item wdt:P18 ?image .
    }
    BIND (REPLACE(wikibase:decodeUri(SUBSTR(STR(?image), 52)), " ", "_") AS ?filename)
    BIND (MD5(?filename) AS ?MD5)
    BIND (URI(CONCAT("https://upload.wikimedia.org/wikipedia/commons/",
                     SUBSTR(?MD5, 1, 1), "/", SUBSTR(?MD5, 1, 2), "/", ?filename)) As ?contentUrl)
  }
} AS %get_some_images_from_Wikidata
WHERE
{
  INCLUDE %get_some_images_from_Wikidata
  OPTIONAL { ?file schema:contentUrl ?contentUrl . }
}

The content URLs are constructed as described in https://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory and all I tested were correct. But the query gives this result:

images	files
6328	1013

So only 1013 out of 6328 files used in Wikidata claims can be found in WCQS.

Event Timeline

Dipsacus_fullonum created this task.Dec 2 2020, 10:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2020, 10:13 PM

Reedy added a project: SDC General.Dec 3 2020, 1:40 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptDec 3 2020, 1:40 AM

Adding Lydia on her request.

Omar_Ghrida added a project: Commons.Dec 3 2020, 12:09 PM

CBogen added a project: Wikidata-Query-Service.Dec 3 2020, 2:44 PM

CBogen moved this task from Incoming to Current work on the Wikidata-Query-Service board.Dec 7 2020, 4:30 PM

CBogen added a project: Discovery-Search (Current work).

RKemper updated the task description. (Show Details)Dec 7 2020, 6:23 PM

Timebox to a half a day investigation and re-evaluate.

CBogen moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.Dec 7 2020, 6:25 PM

• Zbyszko claimed this task.Mar 3 2021, 10:12 AM

• Zbyszko moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.

First M-entity mentioned in the ticket was missing because there was a bug with weekly reloads, that has now been fixed - entries added before the reload should be available.

As for the second part of the ticket - sdc has uri encoded contentUrl (although apparently MD5 is calculated from decoded filename). I modified the query to match that fact:

SELECT (COUNT (DISTINCT ?image) AS ?images) (COUNT(DISTINCT ?file) AS ?files)
WITH
{
  SELECT ?image ?contentUrl
  WHERE
  {
    SERVICE <https://query.wikidata.org/sparql>
    {
      ?item wdt:P31 wd:Q5153359 .
      ?item wdt:P18 ?image .
    }
    BIND (REPLACE(wikibase:decodeUri(SUBSTR(STR(?image), 52)), " ", "_") AS ?filename)
    BIND (REPLACE(SUBSTR(STR(?image), 52), "%20", "_") AS ?filenameUnencoded)
    BIND (MD5(?filename) AS ?MD5)
    BIND (URI(CONCAT("https://upload.wikimedia.org/wikipedia/commons/",
                     SUBSTR(?MD5, 1, 1), "/", SUBSTR(?MD5, 1, 2), "/", ?filenameUnencoded)) As ?contentUrl)
  }
} AS %get_some_images_from_Wikidata
WHERE
{
  INCLUDE %get_some_images_from_Wikidata     
  OPTIONAL { ?file schema:contentUrl ?contentUrl . }
}

This yields (at the time of writing):

images	files
6326	6318

Which leaves 8 unaccounted for. Out of those, 7 do not have structured data defined and last one has a new structured data content, so possibly it wasn't present in the latest dump we reload from.

Please, let me know if this resolves the issue.

• Zbyszko moved this task from In Progress to Waiting on the Discovery-Search (Current work) board.Mar 3 2021, 2:53 PM

• Zbyszko moved this task from Waiting to Blocked/Waiting on the Discovery-Search (Current work) board.Mar 16 2021, 9:40 AM

Looks like this is resolved. If you find more issues, feel free to re-open.

Many files on Commons cannot be found in WCQSClosed, ResolvedPublicBUG REPORTActions

Description

Event Timeline

Many files on Commons cannot be found in WCQS
Closed, ResolvedPublicBUG REPORT
Actions