Maniphest T194884

Get a random set of entities from Wikidata
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	abian
	May 17 2018, 1:45 PM

Project Tags

Referenced Files

	F23242256: random.png
	Jul 2 2018, 9:51 PM

	F23242279: non-random.png
	Jul 2 2018, 9:51 PM

Subscribers

abian

Aklapper

Smalyshev

Description

There are many entities in Wikidata and processing them all is too expensive for certain purposes. However, for statistical purposes (for example, to get any kind of proportion of property use, completeness, consistency, etc.), it's not necessary to retrieve and process them all, a small subset can be enough if representative (random).

Currently, it's hard to retrieve a random data set from Wikidata because:

the Wikidata Query Service doesn't retrieve entities randomly;
Special:Random requires two requests for every retrieved entity (first, a HTTP GET to Special:Random; then, a HTTP GET to the suggested item), doesn't support filters, and offers no significant advantage over directly generating random integers and addressing HTTP requests to the corresponding URIs.

It would be useful to have either:

the possibility of randomly retrieving data through the Wikidata Query Service (best option), or
a new tool to download an arbitrary number of random entities from Wikidata as a single file on demand.

Related Objects

Mentioned In: T211355: Monitor the temporal falsifiability of people in Wikidata

Event Timeline

abian created this task.May 17 2018, 1:45 PM

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptMay 17 2018, 1:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

abian updated the task description. (Show Details)May 17 2018, 1:54 PM

• Vvjjkkii renamed this task from Get a random set of entities from Wikidata to vtcaaaaaaa.Jul 1 2018, 1:09 AM

• Vvjjkkii triaged this task as High priority.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

CommunityTechBot renamed this task from vtcaaaaaaa to Get a random set of entities from Wikidata.Jul 2 2018, 4:05 PM

CommunityTechBot raised the priority of this task from High to Needs Triage.

CommunityTechBot updated the task description. (Show Details)

CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

CommunityTechBot added a subscriber: Aklapper.

How random should it be?
What I mean, when you get a result set from WDQS, without sorting, it is in kind of "random" order already, due to multi-threaded nature of query processing in Blazegraph. But this order is not truly random - it just has some randomness in it due to the internals of the engine, but not suitable for, say, statistical sampling.

Theres's a sample service:

SELECT * WHERE {
service bd:sample {
  ?s wdt:P31 wd:Q5 .
   #
   # optional service params for the sample 
   bd:serviceParam bd:sample.limit 200 .
#   bd:serviceParam bd:sample.seed 0 .
   bd:serviceParam bd:sample.sampleType "RANDOM" .
}
}

That produces 200 random items about humans. Would that work for your use case?

This query...

SELECT ?s WHERE { ?s wdt:P31 ?o } LIMIT 200

... gives us the following distribution of QIDs...

... while this query...

SELECT ?s WHERE {
  SERVICE bd:sample {
    ?s wdt:P31 ?o .
    bd:serviceParam bd:sample.limit 200 .
    bd:serviceParam bd:sample.sampleType "RANDOM" .
  }
}

... gives us the following distribution of QIDs. <3

This is great, @Smalyshev! We could prepare an example query so that people can see this trick (and so that we don't forget it). :-)

abian reassigned this task from abian to Smalyshev.Jul 2 2018, 9:51 PM

abian mentioned this in T211355: Monitor the temporal falsifiability of people in Wikidata.Dec 6 2018, 5:24 PM

Get a random set of entities from WikidataClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Get a random set of entities from Wikidata
Closed, ResolvedPublic
Actions