Page MenuHomePhabricator
Create Task
Maniphest T364363

[Epic] Productionize federated wdqs graph-split endpoints
Closed, ResolvedPublic
Actions

Description

This epic tracks the SRE operational work to get at least a couple wdqs hosts fully in production with the new graph split way of doing things (main vs scholarly graphs).

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT335067 Epic: Wikidata Query Service stabilization
 ResolvedBTracy-WMFT337013 [Epic] Splitting the graph in WDQS
 ResolvedRKemperT364363 [Epic] Productionize federated wdqs graph-split endpoints
 ResolvedRKemperT364364 Provision DNS and certificates for wdqs graph split domains
Resolved• StevemuneneT364366 Implement puppet logic for wdqs graph split
 Resolved• StevemuneneT364367 Create dedicated UIs for wdqs graph split endpoints
 Resolved• StevemuneneT364368 Create separate pybal pools for wdqs graph split (main vs scholarly)
 Resolved• StevemuneneT372919 Bring wqds2024 back into service
 ResolvedRKemperT373145 Create new service catalog entries for wdqs-main and wdqs-scholarly
 ResolvedRKemperT364077 Adapt the wdqs data-transfer cookbook to operate with federated subgraphs
 InvalidGehelT368972 Use Envoy instead of LVS to route internal federation traffic for WDQS
 ResolvedRKemperT370754 Import WDQS subgraphs to production nodes
 ResolvedItamarWMDET371346 Add a banner to query.wikidata.org to inform users about the scholarly articles split
 Resolved• StevemuneneT371833 Tear down wdqs graph split experimental endpoints
 DuplicateNoneT372816 Add federated queries in the test queries used to validate deployment
 ResolvedRKemperT372818 Verify and correct WDQS Grafana dashboard with graph split
 ResolveddcausseT373391 Create wdqs-main and wdqs-scholarly specific test queries
 ResolvedbkingT373791 Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
 ResolveddcausseT373812 Internal federation sometimes fail with HttpConnectionOverHTTP
 ResolvedRKemperT384422 Provide a low availability / scalability full graph endpoint to ease the transition to a split graph
 ResolvedRKemperT392877 Make WDQS legacy full graph endpoint visible on dashboard
 ResolvedRKemperT388134 Drop support for the full Wikidata graph from query.wikidata.org

Event Timeline

RKemper created this task.May 7 2024, 6:25 AM
Gehel moved this task from Current work to Ops / SRE on the Discovery-Search board.May 10 2024, 8:16 AM
Gehel edited projects, added: Discovery-Search; removed: Discovery-Search (Current work).
Gehel added a project: Data-Platform-SRE.
Gehel triaged this task as High priority.May 10 2024, 8:22 AM
Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.
Gehel moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.May 13 2024, 12:47 PM
RKemper added a subtask: T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.Jun 20 2024, 4:03 PM
• Stevemunene closed subtask T364364: Provision DNS and certificates for wdqs graph split domains as Resolved.Jun 26 2024, 12:15 PM
Gehel reopened subtask T364364: Provision DNS and certificates for wdqs graph split domains as Open.Jul 8 2024, 6:41 PM
Gehel closed subtask T368972: Use Envoy instead of LVS to route internal federation traffic for WDQS as Invalid.Jul 9 2024, 10:04 AM
RKemper closed subtask T364367: Create dedicated UIs for wdqs graph split endpoints as Resolved.Jul 16 2024, 5:57 PM
dcausse added a subtask: T371346: Add a banner to query.wikidata.org to inform users about the scholarly articles split.Jul 30 2024, 7:17 AM
dcausse mentioned this in T371871: Very high maxlag on Wikidata due to WDQS lag.Aug 6 2024, 9:30 AM
• Lucas_Werkmeister_WMDE mentioned this in T369079: Update `UniqueValueChecker` to query a list of endpoints.Aug 15 2024, 11:21 AM
Gehel closed subtask T364366: Implement puppet logic for wdqs graph split as Resolved.Aug 21 2024, 9:34 AM
Gehel mentioned this in T364366: Implement puppet logic for wdqs graph split.
dr0ptp4kt mentioned this in T371833: Tear down wdqs graph split experimental endpoints.Aug 23 2024, 10:52 AM
dcausse added a subtask: T373791: Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022.Sep 2 2024, 9:49 AM
dcausse added a subtask: T373812: Internal federation sometimes fail with HttpConnectionOverHTTP.Sep 2 2024, 2:25 PM
• Stevemunene closed subtask T371833: Tear down wdqs graph split experimental endpoints as Resolved.Sep 4 2024, 9:18 AM
• Stevemunene closed subtask T364368: Create separate pybal pools for wdqs graph split (main vs scholarly) as Resolved.
RKemper changed the status of subtask T372818: Verify and correct WDQS Grafana dashboard with graph split from Open to In Progress.Sep 5 2024, 10:01 PM
bking closed subtask T373791: Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022 as Resolved.Sep 11 2024, 5:32 PM
RKemper closed subtask T364364: Provision DNS and certificates for wdqs graph split domains as Resolved.Sep 17 2024, 5:44 PM
Lydia_Pintscher closed subtask T371346: Add a banner to query.wikidata.org to inform users about the scholarly articles split as Resolved.Sep 18 2024, 8:24 AM
Gehel closed subtask T370754: Import WDQS subgraphs to production nodes as Resolved.Sep 20 2024, 9:04 AM
Gehel moved this task from Quarterly Goals to 2024.09.06 - 2024.09.27 on the Data-Platform-SRE board.Sep 24 2024, 2:30 PM
Gehel edited projects, added: Data-Platform-SRE (2024.09.06 - 2024.09.27); removed: Data-Platform-SRE.
Gehel edited projects, added: Data-Platform-SRE (2024.09.28 - 2024.10.18); removed: Data-Platform-SRE (2024.09.06 - 2024.09.27).Sep 27 2024, 1:00 PM
Gehel closed subtask T373391: Create wdqs-main and wdqs-scholarly specific test queries as Resolved.Sep 27 2024, 1:11 PM
RKemper closed subtask T372818: Verify and correct WDQS Grafana dashboard with graph split as Resolved.Oct 1 2024, 2:47 PM
• Stevemunene closed subtask T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs as Resolved.Oct 4 2024, 7:31 AM
BTullis edited projects, added: Data-Platform-SRE (2024.10.19 - 2024.11.08); removed: Data-Platform-SRE (2024.09.28 - 2024.10.18).Oct 18 2024, 3:09 PM
Gehel closed subtask T373812: Internal federation sometimes fail with HttpConnectionOverHTTP as Resolved.Nov 1 2024, 9:53 AM
Gehel edited projects, added: Data-Platform-SRE (2024.11.09 - 2024.11.29); removed: Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
Gehel edited projects, added: Data-Platform-SRE (2024.11.30 - 2024.12.20); removed: Data-Platform-SRE (2024.11.09 - 2024.11.29).Nov 29 2024, 1:23 PM
Gehel edited projects, added: Data-Platform-SRE (2025.01.11 - 2025.01.31); removed: Data-Platform-SRE (2024.11.30 - 2024.12.20).Jan 10 2025, 10:39 AM
Gehel edited projects, added: Data-Platform-SRE (2025.02.10 - 2025.02.28); removed: Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:05 AM
Gehel edited projects, added: Data-Platform-SRE (2025.03.01 - 2025.03.21); removed: Data-Platform-SRE (2025.02.10 - 2025.02.28).Feb 28 2025, 1:27 PM
RKemper closed subtask T384422: Provide a low availability / scalability full graph endpoint to ease the transition to a split graph as Resolved.Mar 10 2025, 7:48 PM
Gehel edited projects, added: Data-Platform-SRE (2025.03.22 - 2025.04.11); removed: Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:57 AM
Gehel edited projects, added: Data-Platform-SRE (2025.04.12 - 2025.05.02); removed: Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM
Gehel edited projects, added: Data-Platform-SRE (2025.05.02 - 2025.05.23); removed: Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM
Gehel moved this task from Backlog - project to Tracking on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 8 2025, 1:47 PM
RKemper closed this task as Resolved.May 14 2025, 7:11 PM
RKemper claimed this task.
RKemper closed subtask T388134: Drop support for the full Wikidata graph from query.wikidata.org as Resolved.
RKemper moved this task from Tracking to Done on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 16 2025, 9:23 AM
Gehel edited projects, added: Discovery-Search (2025.05.02 - 2025.05.23); removed: Discovery-Search.May 16 2025, 12:17 PM
Gehel moved this task from Incoming to Reported on the Discovery-Search (2025.05.02 - 2025.05.23) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits