The table wikibase_rdf contains 4 columns (not counting partition columns):
- context
- subject
- preficate
- object
We should write a job that can converts a given partition into a format that is readable by an RDF compliant application (blazegraph must support this format). The formats used in our infracture are generally Turtle and n3 (more formats esp. faster binary ones can be evaluated but this is out of scope of this task).
The output does not have to keep the same ordering as the original RDF output from wikibase but we might consider keeping the triples attached to an entity grouped together (sort by context).
Ideally we want this format to be extracted as plain file, this task does not imply that the tooling is able to do so but some documentation must be added to define a procedure using existing hdfs tools to extract the file content.
AC:
- a spark job is available and can take a triples table, the desired output format, (optional: the desired chunk size) the location of the output
- documentation on how to extract the RDF chunk files out of hdfs
