Once ready locally with unit-tests, apply the triple-analysis method to bigger data in spark (a day).

Update 19/5/2021

Ran the code on a day (10 May 2021) of data and did some basic analysis. The detailed analysis (along with the code) is in the pdf of the Notebook.

~92% of the daily SPARQL queries are processed successfully. Those that aren't parsed contain some additional prefixes like mwapi. All queries that failed parsing were not checked.

Example Extracted Triples

The Query:

SELECT * WHERE
    {
      ?s wdt:P31/wdt:P279 <_:bn>;
         skos:altLabel "alias"@en.
    }

The extracted triples:

[

TripleInfo(
    NodeInfo(NODE_VAR,s), //subject
    NodeInfo(PATH,<http://www.wikidata.org/prop/direct/P31>/<http://www.wikidata.org/prop/direct/P279>), //predicate
    NodeInfo(NODE_BLANK,bn) //object
), 

TripleInfo(
    NodeInfo(NODE_VAR,s), //subject
    NodeInfo(NODE_URI,skos:altLabel), //predicate
    NodeInfo(NODE_LITERAL,alias@en) //object
)

]

Distribution of Node Types for subject, object, and predicate nodes.

Distribution of the values these nodes contain - combined and per node category (sub/pred/obj) - is in the notebook.

Triples Distribution

Based on Node types

Top 50 triples based on values, but keeping variables and blank nodes obfuscated. Because we can name variables anything, the triple still remains the same.