As a user of the wikidata triples database available in hive I want to have all the triples to be unique so that analysis are more accurate.
Due to how the RDF dumps are generated they may contain duplicates. After discussing with @JAllemandou we agreed to do the deduplication early when importing the data.
AC:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| bump rdf-spark-tools to 0.3.77 | wikimedia/discovery/analytics | master | +2 -2 | |
| Deduplicate wikidata triples | wikidata/query/rdf | master | +56 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | AKhatun_WMF | T286436 Deduplicate triples when loading the wikibase RDF dumps into hive | |||
| Open | None | T289753 Optimize deduplication of triples when loading into wikibase RDF dumps | |||
| Open | None | T289754 Triple level deduplication |
Change 707051 had a related patch set uploaded (by AKhatun; author: AKhatun):
[wikidata/query/rdf@master] Deduplicate wikidata triples
Change 707051 merged by jenkins-bot:
[wikidata/query/rdf@master] Deduplicate wikidata triples
Joseph will suggest an optimization to this task when he is back. For now a simple .distinct() has been done on Spark dataframe to facilitate analysis on Wikidata dumps.
Change 708282 had a related patch set uploaded (by DCausse; author: DCausse):
[wikimedia/discovery/analytics@master] bump rdf-spark-tools to 0.3.77
Change 708282 merged by jenkins-bot:
[wikimedia/discovery/analytics@master] bump rdf-spark-tools to 0.3.77