Page MenuHomePhabricator

Investigate common qualifiers for “unknown value” statement main snaks
Closed, ResolvedPublic

Description

In T244341: Stop using blank nodes for encoding SomeValue and OWL constraints in WDQS, the question arose whether we should use “some value” or “unknown value” terminology. The UI and the data model primer use “unknown value”, but the full data model uses “some value”, and @Jheald has argued that this matches the current use of that snak type in Wikidata better (see T244341#5862287). We’d like to get some number on this, but it doesn’t seem to be possible on the public query service – even for a single property, a query like the following times out:

SELECT ?property (COUNT(*) AS ?count) WHERE {
  ?statement ps:P20 ?unknown.
  FILTER(ISBLANK(?unknown))
  ?statement ?pq ?qualifier.
  ?property wikibase:qualifier ?pq.
}
GROUP BY ?property
ORDER BY DESC(?count)

More generally, we would be interested in the number of qualifiers of statements with “unknown value” in the main snak, aggregated by main statement and qualifier property. In SPARQL, that would be more or less:

SELECT ?mainProperty ?qualifierProperty (COUNT(*) AS ?count) WHERE {
  ?mainProperty wikibase:claim ?p;
                wikibase:statementProperty ?ps.
  ?qualifierProperty wikibase:qualifier ?pq.
  ?subject ?p ?statement.
  ?statement ?ps ?unknown. FILTER(isBlank(?unknown))
  ?statement ?pq ?qualifier.
}
GROUP BY ?mainProperty ?qualifierProperty

(This doesn’t count statements without any qualifiers, but we could probably get those numbers separately for the top properties we’re interested in.) This may be more efficient to query with a different technology, though (Hadoop?).

Event Timeline

dcausse added a subscriber: JAllemandou.

@JAllemandou did some work and could extract some numbers from a dump imported to hadoop:

SELECT ?property (COUNT(*) AS ?count) WHERE {
  ?statement ps:P20 ?unknown.
  FILTER(ISBLANK(?unknown))
  ?statement ?pq ?qualifier.
  ?property wikibase:qualifier ?pq.
}
GROUP BY ?property
ORDER BY DESC(?count)
+------------------------------------+-----+                                    
|s1                                  |count|
+------------------------------------+-----+
|http://www.wikidata.org/entity/P1319|7    |
|http://www.wikidata.org/entity/P17  |5    |
|http://www.wikidata.org/entity/P131 |4    |
|http://www.wikidata.org/entity/P1476|1    |
+------------------------------------+-----+
SELECT ?mainProperty ?qualifierProperty (COUNT(*) AS ?count) WHERE {
  ?mainProperty wikibase:claim ?p;
                wikibase:statementProperty ?ps.
  ?qualifierProperty wikibase:qualifier ?pq.
  ?subject ?p ?statement.
  ?statement ?ps ?unknown. FILTER(isBlank(?unknown))
  ?statement ?pq ?qualifier.
}
GROUP BY ?mainProperty ?qualifierProperty
+------------------------------------+------------------------------------+-----+
|mp                                  |qp                                  |count|
+------------------------------------+------------------------------------+-----+
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P1810|77855|
|http://www.wikidata.org/entity/P123 |http://www.wikidata.org/entity/P1932|56081|
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P459 |29573|
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P3519|24827|
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P3382|15795|
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P352 |9593 |
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P2926|7310 |
|http://www.wikidata.org/entity/P570 |http://www.wikidata.org/entity/P1319|4143 |
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P973 |2233 |
|http://www.wikidata.org/entity/P98  |http://www.wikidata.org/entity/P1932|2192 |
|http://www.wikidata.org/entity/P98  |http://www.wikidata.org/entity/P1545|1630 |
|http://www.wikidata.org/entity/P393 |http://www.wikidata.org/entity/P1932|1356 |
|http://www.wikidata.org/entity/P110 |http://www.wikidata.org/entity/P1932|1334 |
|http://www.wikidata.org/entity/P110 |http://www.wikidata.org/entity/P1545|1302 |
|http://www.wikidata.org/entity/P559 |http://www.wikidata.org/entity/P131 |957  |
|http://www.wikidata.org/entity/P655 |http://www.wikidata.org/entity/P1932|952  |
|http://www.wikidata.org/entity/P655 |http://www.wikidata.org/entity/P1545|879  |
|http://www.wikidata.org/entity/P570 |http://www.wikidata.org/entity/P1326|864  |
|http://www.wikidata.org/entity/P569 |http://www.wikidata.org/entity/P1326|690  |
|http://www.wikidata.org/entity/P5202|http://www.wikidata.org/entity/P1932|627  |
|http://www.wikidata.org/entity/P5202|http://www.wikidata.org/entity/P1545|615  |
|http://www.wikidata.org/entity/P50  |http://www.wikidata.org/entity/P1932|542  |
|http://www.wikidata.org/entity/P569 |http://www.wikidata.org/entity/P1319|384  |
|http://www.wikidata.org/entity/P2093|http://www.wikidata.org/entity/P1545|380  |
|http://www.wikidata.org/entity/P1343|http://www.wikidata.org/entity/P3523|369  |
|http://www.wikidata.org/entity/P571 |http://www.wikidata.org/entity/P1326|336  |
|http://www.wikidata.org/entity/P571 |http://www.wikidata.org/entity/P1319|273  |
|http://www.wikidata.org/entity/P3872|http://www.wikidata.org/entity/P137 |223  |
|http://www.wikidata.org/entity/P3872|http://www.wikidata.org/entity/P580 |206  |
|http://www.wikidata.org/entity/P3872|http://www.wikidata.org/entity/P582 |206  |
|http://www.wikidata.org/entity/P26  |http://www.wikidata.org/entity/P580 |199  |
|http://www.wikidata.org/entity/P816 |http://www.wikidata.org/entity/P817 |192  |
|http://www.wikidata.org/entity/P921 |http://www.wikidata.org/entity/P1545|158  |
|http://www.wikidata.org/entity/P816 |http://www.wikidata.org/entity/P1107|139  |
|http://www.wikidata.org/entity/P2679|http://www.wikidata.org/entity/P1932|137  |
|http://www.wikidata.org/entity/P3383|http://www.wikidata.org/entity/P805 |135  |
|http://www.wikidata.org/entity/P921 |http://www.wikidata.org/entity/P1559|131  |
|http://www.wikidata.org/entity/P625 |http://www.wikidata.org/entity/P828 |126  |
|http://www.wikidata.org/entity/P629 |http://www.wikidata.org/entity/P407 |114  |
|http://www.wikidata.org/entity/P1181|http://www.wikidata.org/entity/P2241|101  |
|http://www.wikidata.org/entity/P2679|http://www.wikidata.org/entity/P1545|100  |

full list here

For future reference the technique used was to manually transpile the sparql into spark joins, see P10540.
Since we did not know what to expect it was hard to assert the validity of the results but results from query one are found in the output of the generalized query.

Thanks a lot! I added labels in P10540#61224. In combination with the total counts from T244341#5920206, my interpretation of the top entries is:

  • Out of the 77859 “described by source: unknown value” statements, 77855 have a “named as” qualifier. In “described by source”, unknown value virtually always means “some value, see ‘named as’”. Other common qualifiers, such as “determination method” and various external identifiers, seem unrelated to the interpretation of the unknown value snak.
  • Out of the 57564 “publisher: unknown value” statements, 56081 have a “stated as” qualifier. In “publisher”, unknown value overwhelmingly means “some value, see ‘stated as’”. (I’m vaguely curious if the choice of “named as” vs. “stated as” has any significance, or if it’s just the result of two different people doing separate imports and happening to pick different but related properties.)
  • Out of the 13772 “date of death: unknown value” statements, 4143 have an “earliest date” qualifier, and 864 have a “latest date” qualifier. In “date of death”, unknown value usually does mean “unknown”, though the qualifiers can sometimes constrain the possible range.
  • Out of the 12792 “given name: unknown value” statements, almost none have any qualifier at all (top entry in P10540 is “applies to part”, 34 times). In “given name”, unknown value means “unknown”. I’m not sure in which case this would be used, by the way – even if the external source only recorded an initial, for example, we have items we can use for that (such as S.) instead of unknown value.
  • Out of the 10054 “date of birth: unknown value” statements, 690 have a “latest date” qualifier, and 384 have an “earliest date” qualifier. In “date of birth”, like in “date of death”, unknown value usually means “unknown”, possibly constrained by qualifiers.
  • Out of the 5398 “place of birth: unknown value” statements, almost none have any qualifier at all (top entry in P10540 is “country”, 42 times). In “place of birth”, unknown value means “unknown”. Those cases with a “country” qualifier should probably just enter the country directly into the statement?
  • Out of the 5397 “inception: unknown value” statements, 336 have a “latest date” qualifier, and 273 have an “earliest date” qualifier. In “inception”, unknown value means “unknown”, as in “date of birth/death”.
  • ...skipping some...
  • On the 1534 “editor: unknown value” statements, there are 2192 “stated as” qualifiers and 1630 “series ordinal” qualifiers. This doesn’t really match up (possibly due to the analyses having run at different times? could somebody have added that many other “editor” statements in between?), but it seems that in “editor”, unknown value typically means “some value, see ‘stated as’”.

In conclusion: “unknown value” actually meaning “unknown value” is still common (especially for “time”-type properties); that said, the most common uses really mean “some known value that doesn’t have an item yet”.

  • "named as" = name given to the subject of the statement
  • "stated as" = how the object of the statement was stated

regarding editors : it looks like some works have multiple editors, none of which have Wikidata items. These ought to be given in separate statements distinguished by "series ordinal".
But it may be that because the statements both have the same 'main value' (or, rather, they both have somevalue for the main value), the software used to add the statements may have coalesced them together. QuickStatements tends to do this, I think.

Note that "some value" is a reasonable UI string to stand for "some value, but the value is not known". But "unknown value" is not a good UI string to indicate "some value, which is known, but hasn't been matched to a Wikidata item".

@Lucas_Werkmeister_WMDE can we close this task or is there anything else that needs to be done?
Is there anything else we need to extract to decide what function name we will use for T245533?
Thanks!

I think the investigation is done, I’m not sure who would make the decision about the name. @Lydia_Pintscher? @Lea_Lacroix_WMDE?

Had a quick chat with Lydia, we’ll go with “some value” for the function.