Maniphest T350464

Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	Nov 3 2023, 10:20 AM

Tags

Referenced Files

None

Subscribers

Description

To validate our hypothesis about a graph split, we need SPARQL endpoints exposing both the split graphs and the full graph so that we can compare query results and performances. We want to expose both the full graph and the split graphs, with the same data (based on the same Wikidata dump) so that comparisons are meaningful.

Actual work will be tracked as sub tasks.

AC:

3 servers are loaded with:
- full graph
- scholarly articles
- Full graph (excluding scholarly articles)
SPARQL endpoints are exposed publicly so that our users can experiment

Details

	Subject	Repo	Branch	Lines +/-
	wdqs graph-split: don't use subdomain	operations/puppet	production	+16 -16
	wdqs graph-split: new trafficserver rewrite rules	operations/puppet	production	+36 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T335067 Epic: Wikidata Query Service stabilization
		Open		None	T337013 [Epic] Splitting the graph in WDQS
		Resolved		Gehel	T350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
		Open		dcausse	T349069 Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers
		Resolved		bking	T347504 WDQS graph split: load data from dumps into new hosts
		Resolved		bking	T349011 Improve data-reload cookbook based on graph split needs
		Resolved		bking	T347505 Prepare new WDQS hosts for graph splitting
		Resolved		RKemper	T351650 Expose 3 new dedicated WDQS endpoints
		Resolved		RKemper	T350465 Load Wikidata split graphs into test servers
		Resolved		dr0ptp4kt	T350106 Implement a spark job that converts a RDF triples table into a RDF file format
		Resolved		RKemper	T354658 Create 3 microsites for wdqs full graph, main graph, & scholarly articles
		Resolved		Dzahn	T355273 ProbeDown - *_experimental_wikidata_org
		Resolved		RKemper	T354661 Generate TLS certs for new WDQS endpoints
		Resolved		RKemper	T354662 Create DNS records for 3 new WDQS endpoints
		Open		None	T354043 Decide the name, domain and logo of WDQS for scholarly articles
		Resolved		RKemper	T355593 Re-generate webserver-misc-apps.discovery.wmnet cergen certificate
		Resolved		Gehel	T355888 Enable cross federation between experimental WDQS endpoints
		Resolved		bking	T352878 Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24

Event Timeline

Gehel created this task.Nov 3 2023, 10:20 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2023, 10:20 AM

Gehel added a subtask: T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers.Nov 3 2023, 10:21 AM

Gehel added a subtask: T347504: WDQS graph split: load data from dumps into new hosts.

Gehel added a subtask: T347505: Prepare new WDQS hosts for graph splitting.

Maintenance_bot added a project: Wikidata.Nov 3 2023, 10:31 AM

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.Nov 3 2023, 10:41 AM

Gehel added a project: Discovery-Search (Current work).

Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.

Gehel added a subtask: T351650: Expose 3 new dedicated WDQS endpoints.Nov 20 2023, 7:41 PM

Gehel closed subtask T347504: WDQS graph split: load data from dumps into new hosts as Resolved.Dec 1 2023, 2:45 PM

dcausse added a subtask: T352878: Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24.Dec 19 2023, 8:44 AM

Gehel closed subtask T350465: Load Wikidata split graphs into test servers as Resolved.Dec 20 2023, 2:49 PM

RKemper updated the task description. (Show Details)Jan 8 2024, 5:11 PM

RKemper removed a subtask: T354658: Create 3 microsites for wdqs full graph, main graph, & scholarly articles.Jan 9 2024, 3:51 PM

bking closed subtask T352878: Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24 as Resolved.Jan 16 2024, 4:59 PM

Change 991091 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs graph-split: new trafficserver rewrite rules

https://gerrit.wikimedia.org/r/991091

gerritbot added a project: Patch-For-Review.Jan 16 2024, 8:18 PM

Change 991091 merged by Ryan Kemper:

[operations/puppet@production] wdqs graph-split: new trafficserver rewrite rules

https://gerrit.wikimedia.org/r/991091

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2024, 8:30 PM

Change 991427 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs graph-split: don't use subdomain

https://gerrit.wikimedia.org/r/991427

gerritbot added a project: Patch-For-Review.Jan 17 2024, 10:38 PM

Change 991427 merged by Ryan Kemper:

[operations/puppet@production] wdqs graph-split: don't use subdomain

https://gerrit.wikimedia.org/r/991427

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2024, 11:30 PM

RKemper added a subtask: T355593: Re-generate webserver-misc-apps.discovery.wmnet cergen certificate.Jan 22 2024, 8:02 PM

Gehel removed a subtask: T355593: Re-generate webserver-misc-apps.discovery.wmnet cergen certificate.Jan 23 2024, 2:01 PM

This should be all done, with the new experimental services accessible at:

https://query-full-experimental.wikidata.org/
https://query-main-experimental.wikidata.org/
https://query-scholarly-experimental.wikidata.org/

Gehel closed this task as Resolved.Jan 26 2024, 8:41 AM

Gehel claimed this task.

Gehel closed subtask T351650: Expose 3 new dedicated WDQS endpoints as Resolved.