⚓ T350465 Load Wikidata split graphs into test servers

Status	Assigned	Task
Open	None	T335067 Epic: Wikidata Query Service stabilization
Open	None	T337013 [Epic] Splitting the graph in WDQS
Resolved	Gehel	T350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
Resolved	RKemper	T351650 Expose 3 new dedicated WDQS endpoints
Resolved	RKemper	T350465 Load Wikidata split graphs into test servers
Resolved	dr0ptp4kt	T350106 Implement a spark job that converts a RDF triples table into a RDF file format

Gehel created this task.Nov 3 2023, 10:25 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2023, 10:25 AM

Gehel triaged this task as High priority.Nov 3 2023, 10:26 AM

Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.

Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.

Gehel added a subtask: T350106: Implement a spark job that converts a RDF triples table into a RDF file format.Nov 20 2023, 7:43 PM

@Gehel Is this a duplicate of T347504?

In T350465#9372880, @bking wrote:

@Gehel Is this a duplicate of T347504?

No, it's not. T347504 is about loading the full data set, T350465 is about loading the split data.

Gehel assigned this task to RKemper.Dec 8 2023, 8:47 AM

Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Data-Platform-SRE.

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.

Progress:

wdqs1024 (wikidata main): 6.6B triples loaded, processing chunk 885/1023
wdqs1023 (scholarly articles): 6.3B triples loaded, processing chunk 851/1023

Load seems to have completed:

wdqs1023: 7.6B triples, load time: 5d,21h
wdqs1024: 7.6B triples, load time: 6d,21h

At a glance the number of triples look sane and the blazegraph logs don't show anything suspect, for extra safety we might have to run a couple queries before calling this a success.

Gehel added a parent task: T351650: Expose 3 new dedicated WDQS endpoints.Dec 15 2023, 9:08 AM

Gehel moved this task from In Progress to Needs Review on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.

Gehel moved this task from Blocked/Waiting to Needs review on the Discovery-Search (Current work) board.

Numbers look correct:

host	graph	# entities	# triples
wdqs1022	full	111,514,880	15,320,277,615
wdqs1023	scholarly articles	41,333,875	7,643,858,078
wdqs1024	wikidata main	70,181,005	7,676,622,674