[Epic] Splitting the graph in WDQS
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Lydia_Pintscher
	May 19 2023, 9:35 AM

Description

In order to stabilize the Wikidata Query Service we are looking into splitting the graph inside Blazegraph into 2 (or potentially more) subgraphs. This ticket is for tracking the investigation into what a sensible split would be, what the consequences are and then making it happen.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wdqs categories: switch to internal-main	operations/mediawiki-config	master	+1 -1
	wdqs-categories: remove extraneous wgCirrusSearchCategoryEndpoint value	operations/mediawiki-config	master	+0 -5

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T335067 Epic: Wikidata Query Service stabilization
Resolved	BTracy-WMF	T337013 [Epic] Splitting the graph in WDQS
Resolved	Gehel	T337018 Refine the definitions of the potential splits
Resolved	• Manuel	T337020 [Analytics] Understand the size and connectedness of Wikimedia-internal concepts in the Wikidata graph
Resolved	AndrewTavis_WMDE	T337021 [Analytics] Find out size of term subgraph
Declined	None	T337407 1st experiment in splitting the WDQS graph
Declined	None	T337406 Setting up a test server exposing an experimental split of the WDQS graph
Resolved	dr0ptp4kt	T347989 Adapt rdf-spark-tools to split the wikidata graph based on a set of rules
Resolved	dcausse	T349519 Determine if IGUANA and TFT would fit our query analysis needs
Resolved	Lydia_Pintscher	T349829 List and document all known public SPARQL endpoints serving wikidata
Resolved	Gehel	T350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
Resolved	dcausse	T349069 Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers
Resolved	bking	T347504 WDQS graph split: load data from dumps into new hosts
Resolved	bking	T349011 Improve data-reload cookbook based on graph split needs
Resolved	bking	T347505 Prepare new WDQS hosts for graph splitting
Resolved	RKemper	T351650 Expose 3 new dedicated WDQS endpoints
Resolved	RKemper	T350465 Load Wikidata split graphs into test servers
Resolved	dr0ptp4kt	T350106 Implement a spark job that converts a RDF triples table into a RDF file format
Resolved	RKemper	T354658 Create 3 microsites for wdqs full graph, main graph, & scholarly articles
Resolved	Dzahn	T355273 ProbeDown - *_experimental_wikidata_org
Resolved	RKemper	T354661 Generate TLS certs for new WDQS endpoints
Resolved	RKemper	T354662 Create DNS records for 3 new WDQS endpoints
Resolved	Gehel	T354043 Decide the name, domain and logo of WDQS for scholarly articles
Resolved	RKemper	T355593 Re-generate webserver-misc-apps.discovery.wmnet cergen certificate
Resolved	Gehel	T355888 Enable cross federation between experimental WDQS endpoints
Resolved	bking	T352878 Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24
Resolved	Gehel	T352538 [EPIC] Evaluate the impact of the graph split
Resolved	AndrewTavis_WMDE	T349512 [Analytics] Collect multiple sets of SPARQL queries
Duplicate	AndrewTavis_WMDE	T350157 [Analytics] Extract a representative sample of SPARQL queries from the query logs
Resolved	Gehel	T337015 review sample of queries to get a better sense of impact of various graph split options
Resolved	Gehel	T349911 Explore the feasibility of using SPARQL federation for scholia queries
Resolved	dcausse	T351819 Create a tool that records and compares a set of sparql query results
Resolved	dr0ptp4kt	T355037 Compare the performance of sparql queries between the full graph and the subgraphs
Resolved	dcausse	T355040 Compare the results of sparql queries between the fullgraph and the subgraphs
Invalid	Sannita	T356773 [tracking] Community feedback for the WDQS Split the Graph project
Resolved	Lucas_Werkmeister_WMDE	T355298 Investigate the impact of the WDQS graph split on constraints checks
Resolved	dcausse	T357966 Document limitations of blazegraph federation
Resolved	dcausse	T357980 Compile a set of queries rewritten with federation across the two graph splits
Resolved	pfischer	T361935 Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs
Resolved	EBernhardson	T361950 Ensure that WDQS query throttling does not interfere with federation
Resolved	dcausse	T362060 Generalize ScholarlyArticleSplitter
Resolved	RKemper	T364363 [Epic] Productionize federated wdqs graph-split endpoints
Resolved	RKemper	T364364 Provision DNS and certificates for wdqs graph split domains
Resolved	• Stevemunene	T364366 Implement puppet logic for wdqs graph split
Resolved	• Stevemunene	T364367 Create dedicated UIs for wdqs graph split endpoints
Resolved	• Stevemunene	T364368 Create separate pybal pools for wdqs graph split (main vs scholarly)
Resolved	• Stevemunene	T372919 Bring wqds2024 back into service
Resolved	RKemper	T373145 Create new service catalog entries for wdqs-main and wdqs-scholarly
Resolved	RKemper	T364077 Adapt the wdqs data-transfer cookbook to operate with federated subgraphs
Invalid	Gehel	T368972 Use Envoy instead of LVS to route internal federation traffic for WDQS
Resolved	RKemper	T370754 Import WDQS subgraphs to production nodes
Resolved	ItamarWMDE	T371346 Add a banner to query.wikidata.org to inform users about the scholarly articles split
Resolved	• Stevemunene	T371833 Tear down wdqs graph split experimental endpoints
Duplicate	None	T372816 Add federated queries in the test queries used to validate deployment
Resolved	RKemper	T372818 Verify and correct WDQS Grafana dashboard with graph split
Resolved	dcausse	T373391 Create wdqs-main and wdqs-scholarly specific test queries
Resolved	bking	T373791 Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
Resolved	dcausse	T373812 Internal federation sometimes fail with HttpConnectionOverHTTP
Resolved	RKemper	T384422 Provide a low availability / scalability full graph endpoint to ease the transition to a split graph
Resolved	RKemper	T392877 Make WDQS legacy full graph endpoint visible on dashboard
Resolved	RKemper	T388134 Drop support for the full Wikidata graph from query.wikidata.org
Resolved	RKemper	T367409 Measure the time to recovery of WDQS after the graph split
Resolved	RKemper	T386098 Run a full data-reload on wdqs-main, wdqs-scholarly and wdqs to capture new blank node labels
Resolved	dcausse	T393713 Regularly reconcile items with delete blank nodes
Resolved	Gehel	T367510 Request permission to create 4 kafka topics in kafka-main (WDQS graph split)
Resolved	Lucas_Werkmeister_WMDE	T369079 Update `UniqueValueChecker` to query a list of endpoints
Resolved	pfischer	T369729 RDF Updater Consumer: Allow "empty" patches
Resolved	RKemper	T372094 WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service
Open	None	T374453 [EPIC] Migration of traffic to new Split Endpoints for WDQS
Resolved	AndrewTavis_WMDE	T377211 Estimate the number of queries needing to migrate away from the full WDQS graph
Resolved	RKemper	T379023 Create WDQS split endpoints for internal traffic and reconfigure clients to use the new endpoints
Resolved	• Stevemunene	T377956 Make WikimediaCampaignEvents use split-graph query service
Resolved	RKemper	T376151 Cutover wdqs-internal to new split endpoints
Resolved	bking	T376150 Prepare hosts to serve wdqs-internal-main & wdqs-internal-scholarly
Resolved	RKemper	T379329 Create puppet config for wdqs-internal-main and wdqs-internal-scholarly roles
Resolved	RKemper	T379334 Create DNS records for wdqs-internal-main and wdqs-internal-scholarly
Resolved	RKemper	T380556 Provision VIPs for wdqs-internal-[main,scholarly]
Resolved	RKemper	T379333 Create envoy config for wdqs-internal-main and wdqs-internal-scholarly
Resolved	RKemper	T379330 Create pybal pools for wdqs-internal-main and wdqs-internal-scholarly
Resolved	• Stevemunene	T374021 Make WikibaseQualityConstraints use split-graph query service
Resolved	RKemper	T380555 Enable LVS for wdqs-internal-[main,scholarly]
Resolved	Gehel	T380594 Inform current wdqs-internal consumers about new internal graph split endpoints
Resolved	dcausse	T388860 Kartotherian should use wdqs-internal-main
Resolved	bking	T374967 wdqs-categories migration: decide where to migrate
Resolved	bking	T376079 eqiad: request 1 VM for wdqs-categories
Resolved	bking	T383257 wdqs-categories migration: explore running categories in docker/k8s
In Progress	None	T384039 Create helm chart for wdqs-categories
Open	None	T384937 Stress-test wdqs-categories
Resolved	RKemper	T395772 Teardown lvs for wdqs public pool
Resolved	RKemper	T405978 Re-image remaining full graph hosts to post-graph-split roles
Invalid	RKemper	T406587 Repeated reimage failures on WDQS hosts
Resolved	bking	T406609 wdqs2017: Apparent hardware issue, rack C2
Invalid	bking	T406617 wdqs1020: Hanging during partitioning step of installation, rack E2
Open	None	T396870 update WDQS UI dashboard to account for graph split

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

bking subscribed.Sep 15 2023, 1:37 PM

~~When T345475 is done, we should have 3 new WDQS hosts in CODFW that could be used for the graph splitting experiment. @RKemper let us know if you have any objections to this plan.~~

We'll use eqiad hosts instead, see T347505

bking added a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 15 2023, 2:02 PM

dr0ptp4kt subscribed.Sep 22 2023, 4:07 PM

bking mentioned this in T347504: WDQS graph split: load data from dumps into new hosts.Sep 27 2023, 3:27 PM

bking added a subtask: T347504: WDQS graph split: load data from dumps into new hosts.

bking removed a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 27 2023, 3:36 PM

bking added a subtask: T347505: Prepare new WDQS hosts for graph splitting.Sep 27 2023, 3:39 PM

Gehel added a project: Discovery-Search (Current work).Oct 17 2023, 9:18 AM

Gehel moved this task from Incoming to Epics on the Discovery-Search (Current work) board.

bking closed subtask T347505: Prepare new WDQS hosts for graph splitting as Resolved.Oct 20 2023, 1:58 PM

dcausse added a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Oct 23 2023, 1:41 PM

dcausse added a subtask: T349519: Determine if IGUANA and TFT would fit our query analysis needs.Oct 23 2023, 2:51 PM

dcausse added a subtask: T349829: List and document all known public SPARQL endpoints serving wikidata.Oct 26 2023, 2:59 PM

dcausse added a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.Oct 27 2023, 2:19 PM

Gehel closed subtask T337018: Refine the definitions of the potential splits as Resolved.Nov 3 2023, 9:55 AM

Gehel removed a subtask: T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers.Nov 3 2023, 10:21 AM

Gehel removed a subtask: T347504: WDQS graph split: load data from dumps into new hosts.

Gehel removed a subtask: T347505: Prepare new WDQS hosts for graph splitting.

dr0ptp4kt closed subtask T347989: Adapt rdf-spark-tools to split the wikidata graph based on a set of rules as Resolved.Nov 20 2023, 5:45 PM

Gehel removed a subtask: T351650: Expose 3 new dedicated WDQS endpoints.Nov 20 2023, 7:41 PM

dcausse added a subtask: T351819: Create a tool that records and compares a set of sparql query results.Nov 22 2023, 2:01 PM

Gehel closed subtask T349519: Determine if IGUANA and TFT would fit our query analysis needs as Resolved.Dec 1 2023, 9:20 AM

Gehel removed a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Dec 1 2023, 2:49 PM

Gehel removed a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.

Gehel removed a subtask: T351819: Create a tool that records and compares a set of sparql query results.

SEgt-WMF subscribed.Jan 8 2024, 6:58 PM

Gehel closed subtask T350464: Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph as Resolved.Jan 26 2024, 8:41 AM

dcausse added a subtask: T357966: Document limitations of blazegraph federation.Feb 20 2024, 11:03 AM

dcausse added a subtask: T357980: Compile a set of queries rewritten with federation across the two graph splits.Feb 20 2024, 2:00 PM

Lydia_Pintscher closed subtask T349829: List and document all known public SPARQL endpoints serving wikidata as Resolved.Mar 4 2024, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:10Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:14Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:18Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:38Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Kanashimi subscribed.Mar 7 2024, 10:37 PM

Gehel closed subtask T357980: Compile a set of queries rewritten with federation across the two graph splits as Resolved.Mar 8 2024, 10:19 AM

dcausse added a subtask: T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs.Apr 5 2024, 12:53 PM

Gehel closed subtask T357966: Document limitations of blazegraph federation as Resolved.Apr 5 2024, 1:01 PM

dcausse added a subtask: T361950: Ensure that WDQS query throttling does not interfere with federation.Apr 5 2024, 3:29 PM

Daniel_Mietchen subscribed.Apr 6 2024, 1:04 AM

dcausse added a subtask: T362060: Generalize ScholarlyArticleSplitter.Apr 8 2024, 11:48 AM

Gehel added a subtask: T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.May 10 2024, 8:27 AM

Gehel closed subtask T362060: Generalize ScholarlyArticleSplitter as Resolved.May 24 2024, 7:28 AM

Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jun 21 2024, 1:17 PM

Gehel reopened subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Open.Jun 21 2024, 1:26 PM

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

Lydia_Pintscher added a subtask: T369079: Update `UniqueValueChecker` to query a list of endpoints.Jul 3 2024, 6:52 AM

In T337013#9923203, @Nosferattus wrote:

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

I don't believe this is using the Query Service. This means it would not be affected.

diegodlh subscribed.Jul 8 2024, 12:34 AM

pfischer changed the status of subtask T369729: RDF Updater Consumer: Allow "empty" patches from Open to In Progress.Jul 11 2024, 8:28 AM

Lucas_Werkmeister_WMDE closed subtask T355298: Investigate the impact of the WDQS graph split on constraints checks as Resolved.Jul 16 2024, 8:28 AM

Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jul 19 2024, 12:42 PM

Gehel closed subtask T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs as Resolved.

Gehel closed subtask T369729: RDF Updater Consumer: Allow "empty" patches as Resolved.

Gehel closed subtask T361950: Ensure that WDQS query throttling does not interfere with federation as Resolved.Jul 19 2024, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:52:57Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:53:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Nosferattus unsubscribed.Aug 23 2024, 4:00 PM

bking mentioned this in T372094: WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service.Sep 4 2024, 2:06 PM

bking added a subtask: T372094: WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service.

bking mentioned this in T374016: Consider separating wdqs-categories from the rest of the wdqs stack.Sep 4 2024, 2:45 PM

Lucas_Werkmeister_WMDE closed subtask T369079: Update `UniqueValueChecker` to query a list of endpoints as Resolved.Sep 4 2024, 3:26 PM

Gehel closed subtask T352538: [EPIC] Evaluate the impact of the graph split as Resolved.Sep 6 2024, 12:52 PM

Gehel removed a subtask: T374021: Make WikibaseQualityConstraints use split-graph query service.Sep 10 2024, 2:06 PM

bking mentioned this in T374967: wdqs-categories migration: decide where to migrate.Sep 23 2024, 2:36 PM

bking mentioned this in T375520: EPIC: WDQS categories migration.Sep 30 2024, 9:35 PM

• Stevemunene closed subtask T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs as Resolved.Oct 4 2024, 7:31 AM

bking added a subtask: T374967: wdqs-categories migration: decide where to migrate.Oct 4 2024, 1:21 PM

dcausse reopened subtask T369079: Update `UniqueValueChecker` to query a list of endpoints as Open.Jan 14 2025, 4:51 PM

Gehel edited projects, added Discovery-Search (2025.02.10 - 2025.02.28); removed Discovery-Search (Current work).Feb 11 2025, 2:20 PM

Gehel moved this task from Incoming to Epics on the Discovery-Search (2025.02.10 - 2025.02.28) board.

bking closed subtask T374967: wdqs-categories migration: decide where to migrate as Resolved.Feb 14 2025, 2:26 PM

Change #1122151 had a related patch set uploaded (by Bking; author: Bking):

[operations/mediawiki-config@master] wdqs-categories: use new split graph hosts (wdqs-main) for categories

https://gerrit.wikimedia.org/r/1122151

gerritbot added a project: Patch-For-Review.Feb 24 2025, 2:46 PM

Gehel edited projects, added Discovery-Search (2025.03.01 - 2025.03.21); removed Discovery-Search (2025.02.10 - 2025.02.28).Feb 28 2025, 1:38 PM

Gehel moved this task from Incoming to Epics on the Discovery-Search (2025.03.01 - 2025.03.21) board.

Change #1124535 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/mediawiki-config@master] wdqs categories: switch to internal-main

https://gerrit.wikimedia.org/r/1124535

Gehel edited projects, added Discovery-Search; removed Discovery-Search (2025.03.01 - 2025.03.21).Mar 5 2025, 10:22 AM

Gehel moved this task from 2025.03.01 - 2025.03.21 to Active Epics on the Discovery-Search board.

karapayneWMDE closed subtask T369079: Update `UniqueValueChecker` to query a list of endpoints as Resolved.Mar 14 2025, 8:38 AM

Change #1122151 merged by Ryan Kemper:

[operations/mediawiki-config@master] wdqs-categories: remove extraneous wgCirrusSearchCategoryEndpoint value

https://gerrit.wikimedia.org/r/1122151

Change #1124535 merged by jenkins-bot:

[operations/mediawiki-config@master] wdqs categories: switch to internal-main

https://gerrit.wikimedia.org/r/1124535

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:26:26Z] <ryankemper@deploy1003> Started scap sync-world: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]]

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2025, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:33:11Z] <ryankemper@deploy1003> ryankemper: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:48:06Z] <ryankemper@deploy1003> Finished scap sync-world: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]] (duration: 21m 40s)